A/B testing

Feb 24, 2018 13 Comments

How do you test the impact of a new website feature (on business metrics) which requires a user to opt into using it? I’m assuming that
A/B testing isn’t an option here due to potential bias in results.
Thanks!

comments

Want to comment? LOG IN or SIGN UP
TOP 13 Comments
  • Microsoft / Eng
    AmlEvil

    Microsoft Eng

    BIO
    Sinful birth under the cross
    AmlEvilmore
    It’s hard to say without knowing more about the feature, but can you have the users opt in, but in one case they get the new feature and in the other case they’ll get nothing?
    Feb 24, 2018 4
    • New
      fustercluk

      New

      BIO
      UX
      fusterclukmore
      This is genius
      Feb 24, 2018
    • Facebook / Eng E12
      This doesn’t solve the biggest problem that those who opt in could be enormously different than those who don’t.
      Feb 24, 2018
    • Microsoft / Eng
      AmlEvil

      Microsoft Eng

      BIO
      Sinful birth under the cross
      AmlEvilmore
      But it sounds like this feature cannot be turned on without the user opting in, so why does it matter what the users who don’t opt in are like?
      Feb 24, 2018
    • Facebook / Eng E12
      That’s probably a reasonable assumption given OPs description. However there are a lot of ‘test’ features that have to be opt in due to changes in the terms of service this requires, but there is still the hope to make a mandatory launch in the future so estimating the LATE is less useful.

      Another issue is that if you ask someone to opt in and you are explicit about the change you mess with their expectations and they are no longer a valid control group. Google solves this by having a small set of users with an experimental/labs opt in where you agree to have expts run on you.
      Feb 24, 2018
  • Facebook / Eng E12
    Are you trying to estimate the effect of rolling this out as mandatory for all users (average treatment effect) or how users who opt in would respond (local average treatment effect)?

    If the former you have to deal with people opting in being fundamentally different than those who don’t (selection bias). You could do some kind of population adjustment with a propensity score, but are in trouble if selection is mostly driven by unobservables.

    If the latter, just take those who opt in and run the standard A/B test design on those users.

    This assumes there is no test control interference between opt in and non opt in users like network effects.
    Feb 24, 2018 2
    • Facebook RandForest
      This is roughly right
      Feb 24, 2018
    • New hHAq67
      @e12, @RandForest Any good sources to read around experiment design and proper evaluation?
      Feb 25, 2018
  • OP
    Thanks for all the responses everyone! Great ideas! To add some more details, I’m trying to answer 2 questions:

    1. How much lift does this feature cause?(opt-in is voluntary)

    2. When rolled out site wide what is the net lift we could expect? (opt-in still voluntary)

    Summarizing my understanding from discussion above, for #2 we could just do a usual A/B test. And site wide impact = % users opted-in * lift per opted-in user (both come from test)

    For #1 however, I think a pre-post on opted in users is better. And there are no network effects.

    Does that sound right?
    Feb 25, 2018 1
    • Facebook Jbsbshg
      Yeah sounds good. If it ends up being too much effort, #1 doesn’t sound like it is worth answering if lift per opted in user is high enough from #2.
      Feb 25, 2018
  • Facebook Jbsbshg
    Do a normal A/B test, where B is just the opt in experience. If like 1% of people opt in then you’re probably screwed, but if a decent amount do opt in, then you can get a directional sense for how things are going to trend if fully rolled out. Besides, wouldn’t that be the experience if you do decide to launch it? Or in launch will there be no opt in?
    Feb 24, 2018 1
    • Facebook Jbsbshg
      To be sure, if the final rollout of the feature will ALWAYS require opt in then your experiment is really straightforward.
      Feb 24, 2018
  • Twitch / Eng xj4z77
    Let’s ask Snapchat with that UI update 😬
    Feb 24, 2018 0

Salary
Comparison

    Real time salary information from verified employees