Computing and Library Services - delivering an inspiring information environment

Improving the Performance of Focused Web Crawlers

Batsakis, Sotiris, Petrakis, Euripides and Milios, Evangelos (2009) Improving the Performance of Focused Web Crawlers. Data and Knowledge Engineering, 68 (10). pp. 1001-1013. ISSN 0169-023X

[img] PDF - Accepted Version
Restricted to Repository staff only

Download (802kB)


This work addresses issues related to the design and implementation of focused crawlers. Several variants of state-of-the-art crawlers relying on web page content and link information for estimating the relevance of web pages to a given topic are proposed. Particular emphasis is given to crawlers capable of learning not only the content of relevant pages (as classic crawlers do) but also paths leading to relevant pages. A novel learning crawler inspired by a previously proposed Hidden Markov Model (HMM) crawler is described as well. The crawlers have been implemented using the same baseline implementation (only the priority assignment function differs in each crawler) providing an unbiased evaluation framework for a comparative analysis of their performance. All crawlers achieve their maximum performance when a combination of web page content and (link) anchor text is used for assigning download priorities to web pages. Furthermore, the new HMM crawler improved the performance of the original HMM crawler and also outperforms classic focused crawlers in searching for specialized topics.

Item Type: Article
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Schools: School of Computing and Engineering
School of Computing and Engineering > High-Performance Intelligent Computing
Depositing User: Sotirios Batsakis
Date Deposited: 20 Sep 2013 10:38
Last Modified: 28 Aug 2021 19:41


Downloads per month over past year

Repository Staff Only: item control page

View Item View Item

University of Huddersfield, Queensgate, Huddersfield, HD1 3DH Copyright and Disclaimer All rights reserved ©