ScholarLens Back to search
2020•Journal of Physics Conference SeriesOpen access

Summary of web crawler technology research

Linxuan Yu, Yeli Li, Qingtao Zeng, Yanxiong Sun, Yuning Bian, Wei He

Open full text 28 citations

Abstract

Abstract With the continuous development of network information technology, there is a large amount of unstructured data called big data on the network. Human resources to collect information laborious, so web crawler technology came into being. This paper explores the basic principle and characteristics of web crawler and the classification of current popular crawler, introduces the key technology of crawler, compares two search strategies and the current application of crawler. Finally, the future research direction of web crawler is introduced.

Open-access reader

About this research paper

What this paper is about

Abstract With the continuous development of network information technology, there is a large amount of unstructured data called big data on the network. Human resources to collect information laborious, so web crawler technology came into being. This paper explores the basic principle and characteristics of web crawler and the classification of current popular crawler, introduces the key technology of crawler, compares two search strategies and the current application of crawler. Finally, the future research direction of web crawler is introduced.

Why it matters

OpenAlex reports 28 citations for this work. Citation counts describe recorded attention and do not establish research quality.

Key contribution

A contribution statement is not available in the OpenAlex record.

Method / approach

Method details are not available in the OpenAlex metadata.

Main findings

Findings are not separately available in the OpenAlex metadata.

Limitations

Limitations are not available in the OpenAlex metadata.

Applications

Application details are not available in the OpenAlex metadata.

Available abstract

Abstract With the continuous development of network information technology, there is a large amount of unstructured data called big data on the network. Human resources to collect information laborious, so web crawler technology came into being. This paper explores the basic principle and characteristics of web crawler and the classification of current popular crawler, introduces the key technology of crawler, compares two search strategies and the current application of crawler. Finally, the future research direction of web crawler is introduced.

Key concepts: Web crawler, Focused crawler, Computer science, World Wide Web, Key (lock), Information retrieval, Web page, Web navigation

Related papers

Related paper 1Related paper 2Related paper 3
Back to paper searchBrowse research topicsOriginal source
Summary of web crawler technology research — Research Paper | ScholarLens