2004•Software Practice and ExperienceRequires access

UbiCrawler: a scalable fully distributed Web crawler

Paolo Boldi, Bruno Codenotti, Massimo Santini, Sebastiano Vigna

Abstract

Abstract We report our experience in implementing UbiCrawler, a scalable distributed Web crawler, using the Java programming language. The main features of UbiCrawler are platform independence, linear scalability, graceful degradation in the presence of faults, a very effective assignment function (based on consistent hashing) for partitioning the domain to crawl, and more in general the complete decentralization of every task. The necessity of handling very large sets of data has highlighted some limitations of the Java APIs, which prompted the authors to partially reimplement them. Copyright © 2004 John Wiley & Sons, Ltd.

About this research paper

What this paper is about

Why it matters

OpenAlex reports 564 citations for this work. Citation counts describe recorded attention and do not establish research quality.

Key contribution

A contribution statement is not available in the OpenAlex record.

Method / approach

Method details are not available in the OpenAlex metadata.

Main findings

Findings are not separately available in the OpenAlex metadata.

Limitations

Limitations are not available in the OpenAlex metadata.

Applications

Application details are not available in the OpenAlex metadata.

Available abstract

Key concepts: Web crawler, Scalability, Java, Computer science, Task (project management), Domain (mathematical analysis), Hash function, Web application