A high-precision forum crawler based on vertical crawling

In this paper, we present a special crawler for Internet forums. Different from general crawler and focused crawler, it can get structured information directly get the most valuable Web resources by utilizing the least system resources, filter the useless information to the maximum extent and finally supply users with high-precision information. This crawler adopts template-based processing method which is to use regular expressions to extract structured information. The URL queue is initialized by URLs set in seeds file and valuable URLs are extracted from Web pages and added into the queue during the crawling process. Once the time of one post is beyond the specified time span or the Web information is unchanged, the crawler can skip it in time to avoid wasting system's resources. Experimental results demonstrate that our crawler can collect real-time forum information more efficiently and precisely than other crawlers.