Integrated query and search of databases, XML, and the web

The amount of information available on-line is proliferating at a tremendous rate. At one extreme, traditional database systems are managing large amounts of structured, well-understood data that can be queried via declarative languages such as SQL. At the other extreme, millions of unstructured Web pages are being collected and indexed by search engines for keyword-based search. Recently, XML—the eXtensible Markup Language—has emerged as a simple, practical way to model and exchange semistructured data across the Internet, without the rigid constraints of traditional database systems. This thesis describes work towards unifying and integrating query techniques for traditional databases, search engines, and XML. First, we describe our contributions to the Lore DBMS for managing semi structured data, focusing on ways to enhance system usability for effective querying and searching. Next, we discuss algorithms and indexing techniques that enable effective keyword-based search over traditional and semistructured databases. We then describe how we have migrated and enhanced our research on semistructured data to support the subtle but important nuances of XML. Finally, we describe a new platform that enables efficient combined querying over structured traditional databases and existing Web search engines.