2011•International Journal of Computer ApplicationsOpen access

A Query based Approach to Reduce the Web Crawler Traffic using HTTP Get Request and Dynamic Web Page

Shekhar Mishra, Anurag Jain, Amit Sachan

Abstract

The functions of Web crawler download information from web for search engine.Web pages changed without any notice.Web crawler has to revisit web site to download updated and new web pages.It is estimated 40% of current web traffic is generated by web crawler.This paper proposes query based approach to inform updates on web site to web crawler using Dynamic web page and HTTP GET Request.Dynamic web page generates HTML based response having list of updates on web site after crawler last visit.Web crawler only visits updated web pages instead of visiting full web sites for updates.Proposed scheme is tested & results show that it is very promising.

Open-access reader

About this research paper

What this paper is about

Why it matters

OpenAlex reports 7 citations for this work. Citation counts describe recorded attention and do not establish research quality.

Key contribution

A contribution statement is not available in the OpenAlex record.

Method / approach

Method details are not available in the OpenAlex metadata.

Main findings

Findings are not separately available in the OpenAlex metadata.

Limitations

Limitations are not available in the OpenAlex metadata.

Applications

Application details are not available in the OpenAlex metadata.

Available abstract

Key concepts: Computer science, Web crawler, World Wide Web, Dynamic web page, Focused crawler, Information retrieval, Static web page, Database