Developing tightly-coupled data mining applications on a relational database system

We present a methodology for tightly coupling data mining applications to database systems to build high-performance applications, without requiring any change to the database software. Introduction Most of the current data mining applications have a loose connection with databases. A majority of them treat database simply as a container from which data is extracted to populate main memory data structures before the main execution begins. This approach limits the amount of data the application can handle effectively. The more database-aware applications use looselycoupled SQL to fetch data records as needed by the mining algorithm. The front-end of the application is implemented in a host programming language, with embedded SQL statements in it. The application uses a SQL select statement to retrieve the set of records of interest from the database. A loop in the application program copies records in the result set one-byone from the database address space to the application address...