Similarity based Automatic Web Search Engine Evaluation

Nowadays, as the usage of Internet has incredibly increased, web search engines become the common approach to find and retrieve needed information. Hence, evaluating search engine quality is a hot topic which attracts many researches' attention. In this paper, we propose a framework named Similarity based Automatic Web Search Engine Evaluation, SAWSEE, to evaluate web search engines. SAWSEE measures the information retrieval effectiveness of web search engines by comparing and voting their returned results, particularly using nDCG metric to rank search engines. SAWSEE compares the search engines' results based on their similarity which is calculated in two consecutive levels, the web page address level and the main content of web page level. To find the similarity of the main content of two search engines' results, SAWSEE utilizes Winnowing algorithm, a well-known and widely-used plagiarism detection method. We compared our method with the results acquired from human assessors' evaluations. The promising comparison shows that SAWSEE provides rankings that are consistent with the rankings resulted from human assessors' evaluations. Hence, the proposed method can be applied in real world environments for evaluation of web search engines.