i saw a glassdoor post with this question and thought i'd ask smart people on blind how they would go about tackling it:
A user satisfaction survey was conducted for two groups for a social media platform. Assume large sample sizes.
Group 1 opted into security features
Group 2: has not opted into security features
It was found that user satisfaction (with the overall app) with group 1 was 30% lower than with group 2.
Why do you think so?
What can we conclude?
Should we recommend eliminating this feature?
I think it's quite obvious that there is a huge sampling bias here. The sampling is not random, and there is an inherent bias for people who might opt into security features that will make them less satisfied with a social media app.
I was told to consider stratified sampling, and I've read up on stratified sampling, but I'm not sure how it really helps us answer the main questions such as "should we recommend eliminating this feature?"
side question - how does stratified sampling help us in A/B testing/experiments? is it just to help us ensure that our samples are truly random/balanced?
It seems like you really needed to have some satisfaction score for users before they had the option, and then see the delta for those users after they were presented with the option to opt-in.