Incremental processing for string similarity join

String similarity join is an essential operation of data quality management and a key step to find the value of data. An incremental processing framework for string similarity join is proposed in this paper. Compared with the batching processing model, it can avoid the heavy time cost and the space cost brought by the duplicate similarity computation among historical strings and is suitable for processing data streams. We implement two algorithms: Inc-Join and Inp-Join. Inc-Join runs on a stand-alone machine while Inp-Join runs on a cluster with Spark environment. The experimental results show that this incremental processing framework can reduce the amount of string matching without affecting the join accuracy. When the data quantity becomes large, Inp-Join can make full use of the advantage of parallel processing and obtain a better performance than Inc-Join.