TEMPLATE EXTRACTION FROM HETEROGENEOUS WEB PAGES

Templates are used by many websites for increasing the productivity of publishing the Web pages. Common templates are used with its contents. The templates provide users to easily access the contents due to its consistent structures. However, for machines, the templates are considered harmful because of its irrelevant terms in the template, so it will degrade the performance of web applications. Thus, template detection and extraction techniques have received a lot of attention to improve the performance of search engines, web application, clustering and classification of web documents. The objective of this paper is to cluster the web documents based on the similarity of underlying template structures in the documents so that the templates for each cluster are extracted simultaneously. While extracting the templates, here consider the web page structure along with its contents. To effectively manage an unknown number of clusters (templates) Minimum Description Length (MDL) Principle is use and MinHash technique to estimate the MDL cost quickly. So that it will form a qualified cluster.