Understanding and dealing with operator mistakes in internet services

Abstract Operator mistakes are a significant source of unavailabil-ity in modern Internet services. In this paper, we first characterize these mistakes by performing an extensiveset of experiments using human operators and a realistic three-tier auction service. The mistakes we observedrange from software misconfiguration, to fault misdiagnosis, to incorrect software restarts. We next proposeto validate operator actions before they are made visible to the rest of the system. We demonstrate how toaccomplish this task via the creation of a validation environment that is an extension of the online system, wherecomponents can be validated using real workloads before they are migrated into the running service. We show thatour prototype validation system can detect 66 % of the operator mistakes that we have observed. 1 Introduction Online services, such as search engines, e-mail, work-group calendars, and music juke-boxes are rapidly becoming the supporting infrastructure for numerous users'work and leisure. Increasingly, these services are comprised of complex conglomerates of distributed hardwareand software components, including front-end devices, numerous kinds of application packages, authenticators,loggers, databases, and storage servers.