Research on the development and challenges of dataflow processing technology

Nowadays, unbounded and disordered large-scale data sets are becoming more and more common, and consumers' processing requirements for these data sets are also becoming more and more complex, such as time semantics, windows and processing delays. In order to meet the continuous development of data processing requirements on unbounded and disordered large-scale data sets, this paper analyzes the advantages of dataflow processing from the aspects of low latency, high parallelism, low synchronous memory access cost and simple on-chip logic. On the one hand, the dataflow diagram embodied in the data flow calculation model in big data processing is analyzed from the execution engine level. On the other hand, several typical dataflow processing architecture models are analyzed from the perspective of dataflow batch processing. On this basis, the problems faced in the development of current dataflow are analyzed, and the possible development direction in the future is proposed.