SMART CRAWLER: A TWO-STAGE CRAWLER FOR EFFICIENTLY HARVESTING DEEP-WEB INTERFACES

As wide area of web grows at a very fast pace, there has been increased interest in techniques that help efficiently locate wide web interfaces. However, due to the large volume of web resources and the dynamic nature of deep web, achieving large coverage and high efficiency is a challenging issue. We propose a two-stage framework, namely Smart Crawler, for efficient harvesting wide web interfaces. In the first stage, It is site based searching for center pages with the help of search engines, it avoid to visit large number of pages. To achieve more accurate results for a focused crawl, It is ranking the websites to prioritize highly relevant ones for a given topic. In the second step, It searches fast in-site searching by extracting most relevant links with an adaptive link-ranking. To eliminate bias on visiting some it also contain highly relevant links in hidden web directories, we design a link tree data structure to achieve wider coverage for a website.