An Effective Forum Crawler

Web Forums or Internet Forums provide a space for users to share, discuss and request information. Web Forums are sources of huge amount of structured information that is rapidly changing. So crawling Web Forums require special softwares. A Generic Deep Web Crawler or a Focused Crawler cannot be used for this purpose. In this paper, we propose an effective Web Crawler especially for Internet Forums. This Forum Crawler overcomes the drawbacks of many of the existing Forum Crawlers. It has the ability to detect the Entry URL of a Forum site, given any page of it. Crawling process starting from Entry URL increases the coverage. Different URLs in the Web Forums are classified into four categories and our Forum Crawler is capable of detecting these URLs even if they are JavaScript-based which most of the existing Forum Crawlers cannot do. The entire process is divided into learning part and online crawling part. The learning part classifies different URLs in the forum site into four categories: Index URL, Thread URL, Index-Page-Turning URL and Thread-Page-Turning URL. This Forum Crawler uses a Freshness First Strategy rather than the BFS (Breadth First Strategy) for performing online crawling which is advantageous in situations where there are limited system resources available.