Traditionally, A/B testing is used to compare the control group with one treatment group under some very strict assumptions, most important among them being SUTVA (stable unit treatment value assumptions). It’s not always possible to have the ideal conditions to implement A/B testing in real world applications. Many companies have come up with a version of the test, which still provides statistically significant results.
Netflix is doing massive amounts of experimentation on their platform to improve user experience. One of technique Netflix uses is called interleaving A/B testing design. Instead of showing results from different algorithms to randomly selected users, it shows a blended effect of multiple algorithms to the same user. The position of a recommendation due to a particular algorithm is decided randomly for each user. Though, this involves careful experimental setup but also allows Netflix to trim down the number of options quickly and then apply classical A/B testing to decide on the best algorithm.
LinkedIn like any other social media company has to be vary of network effects while conduction statistical testing on their platform, specially network effects. In order to avoid such pitfalls, LinkedIn decided to change the experimental units in their testing. In order to account for the network effects of their platform, LinkedIn decided to divide their users into market segments. These segments could be based on location (companies conduct experiments separately for each country), age or mutual connections. the segments are such that there is a minimal probability of anyone from segment one networking with someone from segment two in a significant manner. Thus, these segments form an isolated units, preventing the violation of SUTVA.
There are cases when it is simply unavoidable to remove interference or network effects. In those cases, we can implement some quasi-experimental techniques like Differences-in-Difference and Switch-back (time-series) experiments. More on these in my next article.
References
