2022•Unpublished venueRequires access
Research on Anti-crawler and Anti-Anti-crawler Technology
Xiaolong Zhang, Yuan Chen, Jiawei Liang, Jiawei Zhang, Tianyang Lu, Chuanqing Wang
Open publisher page 3 citations
Abstract
In the era of big data, the extensive application of crawler technology makes the operation of the website difficult, and the anti-crawler technology of the website also emerges as the times require. How to develop anti-anti-crawler technology to effectively capture data according to anti-crawler Technology is of great significance for obtaining data. This paper introduces the basic architecture process and corresponding coping strategies of the commonly used website anti-crawler technology and crawler, analyzes the technical methods of anti-crawler, and puts forward some suggestions for anti-anti-crawler technology.