Growth initiatives without A/B testing?
I am of the opinion that A/B testing doesn't move the needle enough. And that as a PM, I would rather spend my time figuring out user needs, or diving into retention and usage patterns to find the features we should be building. However, most companies are like 'A/B testing background is a must'. Why do you think that is the case? What am I missing?
Hypothesis testing is the key, A/B testing is just a means but quite a solid one. At the end of the day you want to learn what works and what doesn't at scale thus driving growth. If you can test 100 hypothesis in a month it doesn't matter what means you used. Having said that, A/B testing helps provide a framework to test these hypothesis at scale.
I don’t think you’re thinking about A/B testing the right way. You A/B test to measure and understand the impact of one design vs another. Any time you do a redesign or build a new product, you conduct an A/B test – there’s a hypothesis you have, and a control and treatment group. You ALWAYS run it as an experiment first to understand user behavior. Are users who are going through the new experience actually being retained more? Are they actually completing the flow more, compared to the old experience? The more testing you do, the more you learn about actual user behavior and how your products impact people. If you’re saying that “A/B testing doesn’t move the needle enough”, you’re not building meaningful products or your hypotheses are repeatedly being proven false.
A/B testing is the most expensive way to disagree.
Sounds like you’re a bigger fan of contextual inquiry.
Definitely not against a/b testing but finding it absolutely ridiculous when the first question a hiring manager of a big company asks is 'how many a/b tests do you run every week'. I am not sure how that correlates with the quality of a PM
Quantity of tests you run certainly doesn’t relate to quality of PM, but it does equate to experience. Sounds like you’re not experienced enough if you haven’t learned anything from A/B testing to think that’s it’s not valuable.
As a B2B PM, if will be hard for you to do A/B tests especially if you have few users. Making the jump from B2B or platform/tools PM to consumer is hard for this reason. So you have to show interviewers how your experience can translate to consumer side, understand deeply how to measure impact more scientifically than you described, and know A/B testing inside and out even though you haven't run any.
It’s a combination of both. Retention and usage patterns are the analysis you do to figure out if product is working. When you want to make changes, you do not know if your idea will work or if there is something different you can do and hence A/B testing
True that! I guess what I meant was I do the first part, find new idea to implement, and then launch to the entire user base (b2b used by one person in the company, hence only a few thousand users). If the feature shows no improvement in the metric, I turn the switch off and remove the feature. In this case, what can I do to be more aligned with a/b testing fundamentals?
What you said aligns with AB testing, the only difference being... a) You wouldn't want to lose a user base by testing a new feature on the entire population, b) you need to be sure that the difference in metric observed is significant and not by chance. AB testing gives a way to sample the population, and test the hypothesis to ensure it is actually the change in feature that brought about the change in metric while controlling for the other factors that might affect it.