2013•International Journal of Computer ApplicationsOpen access

Web Crawler: A Review

Md. AbuKausar, Vijaypal Singh Dhaka, Sanjeev Kumar Singh

Abstract

Information Retrieval deals with searching and retrieving information within the documents and it also searches the online databases and internet.Web crawler is defined as a program or software which traverses the Web and downloads web documents in a methodical, automated manner.Based on the type of knowledge, web crawler is usually divided in three types of crawling techniques: General Purpose Crawling, Focused crawling and Distributed Crawling.In this paper, the applicability of Web Crawler in the field of web search and a review on Web Crawler to different problem domains in web search is discussed.

Open-access reader

About this research paper

What this paper is about

Why it matters

OpenAlex reports 98 citations for this work. Citation counts describe recorded attention and do not establish research quality.

Key contribution

A contribution statement is not available in the OpenAlex record.

Method / approach

Method details are not available in the OpenAlex metadata.

Main findings

Findings are not separately available in the OpenAlex metadata.

Limitations

Limitations are not available in the OpenAlex metadata.

Applications

Application details are not available in the OpenAlex metadata.

Available abstract

Key concepts: Web crawler, Computer science, World Wide Web, Focused crawler, Information retrieval, Web page, Web development, Static web page