Evaluation of Online Personalization Systems: A Survey of Evaluation Schemes and a Knowledge-Based Approach

ABSTRACT The use of personalization techniques for structuring online interactions with customers is common today. In a world in which a large part of customer interactions are done in this manner, systematic evaluation of these methods is critical. In this paper we study the problem of evaluating online personalization, and point out the difficulties of a scientific evaluation from automatically tracked data gathered by most online firms. A factor that contributes to the difficulty is that conducting true experiments may often not be possible due to the potential costs in doing so. When such true experimentation is not possible we present a systematic approach for evaluation that is based on bringing domain knowledge explicitly into the process. The advantage of our approach is that it presents a systematic approach to evaluate these systems by making explicit the domain knowledge. There is indeed no free lunch, and the disadvantage of the approach is that it relies on the accuracy of such domain knowledge. More generally this paper suggests that there may be problems in how online personalization systems are evaluated, and argues for systematic approaches to this important problem. Keywords: Electronic Commerce; Personalization; Evaluation 1. Introduction Businesses are investing considerable resources in developing and deploying personalization systems for customer interactions. Some cited reasons for doing so include a desire to develop stronger relationships with customers, improve profit margins and increase cross-selling [Schonberg et al. 2000, Cutler and Sterne 2000]. In the online world personalization methods are appealing for two key reasons: (1) their ability to differentiate users on a finer basis due to the capacity to collect more data and (2) their ability to do so in a scalable manner due to the automated nature of the techniques. According to a survey of 900 e-business executives conducted by Cahners In- Stat Group, large and medium-sized businesses nearly doubled their use of personalization technologies (for a list of personalization technologies, please see Pal and Rangaswamy, 2003) during 2003, with about 43% already having it in place by 2002. Global investment in personalization technologies is expected to reach $2.1 billion by 2006, up from $500 million in 2002, according to a recent report from Datamonitor. There are several examples of personalization engines in use today. Amazon. com's personalization system, based on collaborative filtering, is well known. AT&T WorldNet1 uses a method to decipher visitors' preferences unintrusively, and continually makes suggestions to visitors based on learned preferences. DoubleClick uses visitor profiles to target banner advertisements on their clients' sites that are more likely to be of interest to a specific visitor. YesMail specializes in targeting and sending personalized emails regarding special deals. Palm uses a recommendation engine that renders graphical product suggestions based on visitors' location on the site. In the business-to-business space, Dell Computer provides personalized Web pages for its corporate customers that simplifies placing and tracking orders. As the above examples suggest, the use of personalization techniques for structuring online interactions with customers is common today. Three factors that enabled this are (1) the extremely large amount of user behavior data tracked at online sites, (2) the availability of powerful personalization techniques in commercial CRM systems, and (3) the ease of implementing interaction strategies on the Web. These factors make it easy to build dynamic models (which can be implemented in real time) that automate interactions with customers. However, it is important to note that this ease of implementing personalized interactions online also raises the possibility of rolling out models with limitations that have not been understood and accounted for. …