A Reference Architecture for Big Data Solutions

From the thesis: This thesis describes a research project with the goal of creating a reference architecture for big data solutions. Big data is an evolution of the field business intelligence and at the same time a revolution in terms of the business value it can bring to organizations. Cloud computing and other inventions make massive parallel processing of data across a large amount of commodity PCs possible. Following the big data breakthroughs, the field of predictive analytics has received a boost, since boundaries of performance and costs have dropped significantly. Thanks to big data technology, organizations can now register, combine, process and analyze data to answer questions that perceived unsolvable a few years ago. An important part of the big data realm is open data. Anyone can obtain or access these data sources directly from the internet, ready to be combined with enterprise data. Useful predictions are possible by combining the internal data of an organization to open data and linking the datasets in a meaningful way. Making the right predictions is only possible when organizations choose the right technology. All the technology options call for a reference architecture that provides guidance to architects for creating big data solutions. This solution reference architecture is an abstraction of 'real' solution architectures. It aims to give guidance to organizations that want to innovate using big data technology, open data sources, and predictive analytics mechanisms for improving their performance. The purpose of the reference architecture is to help with setting up a concrete architecture for big data solutions. The Big Data Solution Reference Architecture was developed and evaluated with one iteration of Hevner’s Information Systems Research Framework. Angelov’s framework for analysis and design of software reference architectures guided the creative design process. An extensive literature study and a qualitative research study using grounded theory on transcribed interviews with big data experts forms the basis of the theoretical model. The resulting reference architecture consists of an abstract diagram of components and interfaces, two architectural patterns, two architecture principles, and two architectural best practices. Ten big data experts evaluated the final reference architecture by answering a questionnaire that measured several quality criteria. Their answers give the indication that the created model is a reasonably good reference architecture for big data solutions, with good practical usability. This model is of scientific and non-scientific importance, since it is be the first empirically reviewed solution reference architecture for big data technology.