SUNVE: Distributed Message Middleware towards Heterogeneous Database Synchronization

With the rapid development of the Internet, the traditional stand-alone database cannot meet the needs of massive log storage and high concurrent users' access, so the distributed database came into being. The distributed database is faced with the challenge of data inconsistency due to the complexity of the local database that makes up the distributed database, such as physical location decentralization, different operating systems on the underlying layer and the different ISPs on the network, so this paper designs a distributed message middleware to synchronize heterogeneous database. The paper presents a method of capturing incremental data based on triggers and log tables to get dynamic incremental data after DML operations on the source database. The paper presents a reliable and real-time message transmission mechanism to guarantee the reliability of the message, namely the order consistency and the exactly-once semantics in the process of message transmission. The paper presents a method of generating full data based on real-time computation of incremental data to achieve the high scalability of the destination database and the high availability of the source database service. It can be seen that the paper designs a highly message-oriented reliable, highly architecture-oriented scalable and highly service-based available distributed message middleware to realize heterogeneous databases synchronization. Based on this distributed middleware, this paper designs and implements a prototype system named SUNVE that supports synchronization service from Oracle to PostgreSQL. The final test shows that: SUNVE strictly guarantees the data consistency between the destination database and the source database. The message transmission of SUNVE is the timeliest and its speed is nearly 2 times to that of Kafka and RocketMQ.