Rarity-Expertise Valuation: The Role of Domain Knowledge in the World of Big Experimentation

Most of the conceptualizations of how big data will change behavioral sciences have been focused on the question of what the world of big data can offer to the behavioral sciences. The recent deployment of industry-level behavioral experimentation, however, prompts the reverse question: what can behavioral science offer to the world of big data in general, and to big experimentation in particular. In this paper we contrast insights from behavioral theories to insights from large-scale experimentation and propose a simple cost-benefit framework that can help evaluate the potential contribution of behavioral science. Based on this framework, we isolate cases where behavioral theories can provide clear benefits from cases where naive approaches alone would perform equally well. While our model is primarily focused on behavioral science, it is also relevant to other fields where big experimentation platforms will become widely used.