Enhanced data extraction, transforming and loading processing for Traditional Chinese Medicine clinical data warehouse

Clinical data warehouse has been developed as a fundamental data infrastructure for large scale TCM clinical data management and decision support services. However, as a key component, data extraction, transforming and loading (ETL) is a complicated and labor intensive task to ensure high data quality before all kinds of data analyses. This paper introduces an enhanced ETL technique framework, which includes operational data store (ODS) model and two step data preprocessing subcomponents, to perform the ETL tasks. The ODS data model was designed to integrate the heterogeneous clinical data sources and support the direct copy from these data sources to ODS database by ETL. Therefore, ETL task has been separated into two core steps in enhanced ETL component: (1) dynamic filter and copy of the original operational data sources to ODS; (2) specialized transforming the ODS data to detailed clinical data warehouse. This enhanced technique framework improves the ETL performance to be used in clinical data center since there would have various kinds of operational data sources that need be integrated in this data environments. This paper has a description of the related enhanced ETL framework and proposes some key procedures to accomplish the tasks.