Design and Implementation of a High-Performance Distributed Web Crawler. Export

@inproceedings{citeulike:2447568, abstract = {Broad web search engines as well as many more specialized search tools rely on web crawlers to acquire large collections of pages for indexing and analysis. Such a web crawler may interact with millions of hosts over a period of weeks or months, and thus issues of robustness, flexibility, and manageability are of major importance. In addition, I/O performance, network resources, and OS limits must be taken into account in order to achieve high performance at a reasonable cost. In this paper, we describe the design and implementation of a distributed web crawler that runs on a network of workstations. The crawler scales to (at least) several hundred pages per second, is resilient against system crashes and other events, and can be adapted to various crawling applications. We present the software architecture of the system, discuss the performance bottlenecks, and describe efficient techniques for achieving high performance. We also report preliminary experimental results based on a crawl of 120 million pages on 5 million hosts.}, author = {Shkapenyuk, Vladislav and Suel, Torsten}, booktitle = {Proceedings of the 18th International Conference on Data Engineering (ICDE)}, citeulike-article-id = {2447568}, citeulike-linkout-0 = {http://cis.poly.edu/tr/tr-cis-2001-03.pdf}, keywords = {crawler, design, web20}, pages = {357--368}, posted-at = {2008-02-29 12:26:08}, priority = {2}, publisher = {IEEE CS Press}, title = {Design and Implementation of a High-Performance Distributed Web Crawler.}, url = {http://cis.poly.edu/tr/tr-cis-2001-03.pdf}, year = {2002} }

TY - CONF ID - citeulike:2447568 L3 - citeulike-article-id:2447568 N2 - Broad web search engines as well as many more specialized search tools rely on web crawlers to acquire large collections of pages for indexing and analysis. Such a web crawler may interact with millions of hosts over a period of weeks or months, and thus issues of robustness, flexibility, and manageability are of major importance. In addition, I/O performance, network resources, and OS limits must be taken into account in order to achieve high performance at a reasonable cost. In this paper, we describe the design and implementation of a distributed web crawler that runs on a network of workstations. The crawler scales to (at least) several hundred pages per second, is resilient against system crashes and other events, and can be adapted to various crawling applications. We present the software architecture of the system, discuss the performance bottlenecks, and describe efficient techniques for achieving high performance. We also report preliminary experimental results based on a crawl of 120 million pages on 5 million hosts. EP - 368 TI - Design and Implementation of a High-Performance Distributed Web Crawler. PB - IEEE CS Press T2 - Proceedings of the 18th International Conference on Data Engineering (ICDE) SP - 357 KW - crawler KW - design KW - web20 AU - Shkapenyuk, Vladislav AU - Suel, Torsten PY - 2002/// UR - http://cis.poly.edu/tr/tr-cis-2001-03.pdf ER -