2015•Indian Journal of Science and TechnologyOpen access

Design of Web Crawler for the Client - Server Technology

Mohammad Abu Kausar, Vijaypal Singh Dhaka, Sanjeev Kumar Singh

Abstract

Search engines store information locally with the purpose of deliver quick, accessible search abilities. This information is collected by Web crawler. Web crawling is necessary for the maintenance of complete and latest web document gathering for a web search tool. The web document modifies its content on regular basis hence it becomes necessary to build up a successful framework which could identify these sorts of changes proficiently in the most minimal scanning time to accomplish this modifications. The essential thought behind designing of such a web crawler is to find high quality web documents within limited time frame. The proposed system works on the Client-Server Technology it reduces the overlap problem and downloads high quality web pages. We can add many web crawler in parallel to download web page in parallel way.Keywords: Client-Server Technology, Overlap, Search Engine, Web Crawler, Web Page

About this research paper

What this paper is about

Why it matters

OpenAlex reports 2 citations for this work. Citation counts describe recorded attention and do not establish research quality.

Key contribution

A contribution statement is not available in the OpenAlex record.

Method / approach

Method details are not available in the OpenAlex metadata.

Main findings

Findings are not separately available in the OpenAlex metadata.

Limitations

Limitations are not available in the OpenAlex metadata.

Applications

Application details are not available in the OpenAlex metadata.

Available abstract

Key concepts: Web crawler, Computer science, Focused crawler, Static web page, World Wide Web, Web page, Web search engine, Web server