2017•International Journal of Advanced Research in Computer ScienceOpen access

Web Crawler : Review of Different Types of Web Crawler, Its Issues, Applications and Research Opportunities

Keyur Desai, Virala Devulapalli, Smita Agrawal, Preeti Kathiria, Atul Patel

Abstract

Today's search engines are equipped with dedicated agents known as “web crawlers” keen to crawling large web contents online which are analyzed and indexed and make the content available to users. Crawlers act together with thousands of web servers over periods expanding from weeks to several years. These crawlers visits several thousands of pages every second, includes a high-performance fault manager, are platform independent or dependent and are able to get used to a wide range of configurations without including additional hardware. This paper is focused on prerequisites of crawler, process of crawling and different types of crawlers. This paper give review about some potential issues related to crawler, applications and research area of web crawler. Keywords: search engine, web crawler, www, Indexing, website analysis,

About this research paper

What this paper is about

Why it matters

OpenAlex reports 22 citations for this work. Citation counts describe recorded attention and do not establish research quality.

Key contribution

A contribution statement is not available in the OpenAlex record.

Method / approach

Method details are not available in the OpenAlex metadata.

Main findings

Findings are not separately available in the OpenAlex metadata.

Limitations

Limitations are not available in the OpenAlex metadata.

Applications

Application details are not available in the OpenAlex metadata.

Available abstract

Key concepts: Web crawler, Focused crawler, Computer science, World Wide Web, Crawling, Web search engine, Search engine indexing, Web page