Controlled experiments on the Web

Ronny Kohavi of Microsoft (previously Amazon) presented a paper this year at KDD called Practical Guide to Controlled Experiments on the Web. As far as I know, it’s the first “academic” paper on what’s often called A/B testing. I say “academic” in quotes because the paper is relatively lightweight and is geared towards an audience of industry practitioners.

Most people who work on A/B testing are computer scientists who know more about systems and databases than statistics, and unfortunately this paper doesn’t do much to correct that. (And by statistics I mean a specific body of knowledge that has been accumulated over the last couple hundred years, not psuedo-scientific Web 2.0 marketese like “long tail” or gratuitous name dropping involving Gauss, Bayes, and Pareto.) However, the paper does point out some system design and usability issues that Amazon and others have learned from their experience. For example, to maintain a consistent user experience, each user must be assigned to the same experimental group on multiple visits to the site. Since maintaining state under distributed servers introduces scaling and performance issues, group assignment based on user ID hashing is the preferred approach.

Given the lack of good technical publications on doing controlled experiments on the Web, this paper is certainly a welcome start.