Starting from below chart, how would you go about calculating an overall retention rate per month? i.e. across all sign-up months, what is the %active after 1m, 2m etc. I was thinking of simply averaging the rates but you get inconsistencies e.g. after 10m you get 16.5% but 17% after 11m (the % should only go down)
What is this?😦
Cohort analysis. The idea is, instead of measuring everything in aggregate across all users, filter out past product performance that not all users were exposed to and look at users in slices... what did that slice think of a product change. How are newer users behaving with the product you’re trying to grow than the people who appear to be sticking around and what does that mean to your long term growth if you might have only initially captured a niche market. It also helps you understand long term impact better... if you only measure retention rate month over month across all users but you traditionally see a drop off after say, month 4, you are probably going to see roughly the same retention rate every month you measure it measured across your whole user base unless you do something drastic, but you never would’ve identified that cliff in month 4 without looking at a view like this, and then you wouldn’t be looking at what to do about it.
Very interesting. Thanks for explanation.
What about going back to the raw data and using cumulative numbers instead? i.e. for period 7 in attached picture, it'd be total # of users who unsubscribed after 7 months or earlier over total # of subscribers (ever)? This way, denominator is constant and numerator increases after each period
Wouldn’t denominator (total subscribers ever) also increase as new cohorts were added rather than stay constant? Either way, you’re going to run into the same types of issues that cohort analysis tries to filter out unless you’re seeing a consistent drop after month 7 and just want to quantify the whole thing across cohorts. You’ve probably made significant changes to the product each month, the competition has probably shifted, and there may be other factors that are going to contribute to differences in behavior across cohorts that would influence behavior and lead to different results for different reasons across cohorts. One thing I do think you can realistically do is, if you didn’t make major product changes for multiple months and the market didn’t change significantly, you could probably safely combine multiple monthly cohorts as long as their numbers were fairly similar. You would just want to acknowledge that the product changes you made over that timeframe, if any, had a near zero impact.
Multilevel survival analysis with dummy variables for cohort?
The point of a cohort view as is shown is that each timeframe is a unique set of users that started using a product at a certain point in the product’s evolution. If the product changes in a positive way, retention should go up in month 1 for new users and month x for users who signed up in prior months. Because product changes can cause both negative and positive results in retention, averaging out retention by duration of how long they’ve been a customer is always going to give the inconsistencies you’re talking about because unless the product has been static all that time and there’s no seasonality, then retention results will wax and wane. I wouldn’t try to do what you’re trying to do with the numbers... it doesn’t make sense to measure in aggregate that way unless you’re looking for vanity metrics.
You could instead look at how a particular product change impacted overall retention in a given month... then you’re just looking at what happened after a change in month 1 for the latest cohort, and every cohort that started before the most recent cohort is the change that happened in the latest month for that cohort as opposed to month 1. Combining all of those together though adds a lot of noise from people who have been with the product longer and may just naturally drop off, people who were influenced by being there longer, etc. IMO it’s better to just look at each cohort individually after a product change and if one looks out of whack vs. the others, try to figure out why.