2003•Unpublished venueRequires access

A Survey on Web Crawling

Sheng Li

Abstract

As a basic component of search engine and a series of other services on Web,Web crawler is playing an important role. Roughly,a Web crawler is a program which automatically traverses the Web by downloading documents and following links from page to page. This article detailedly explains the principles and difficulties on the Web crawler, comprehensively argues several hot directions of Web crawler,and at last views the new direction of Web crawler.

About this research paper

What this paper is about

Why it matters

OpenAlex reports 3 citations for this work. Citation counts describe recorded attention and do not establish research quality.

Key contribution

A contribution statement is not available in the OpenAlex record.

Method / approach

Method details are not available in the OpenAlex metadata.

Main findings

Findings are not separately available in the OpenAlex metadata.

Limitations

Limitations are not available in the OpenAlex metadata.

Applications

Application details are not available in the OpenAlex metadata.

Available abstract

Key concepts: Web crawler, Computer science, Focused crawler, World Wide Web, Web page, Crawling, Web search engine, Upload