Parallel crawlers Export

@inproceedings{citeulike:1586892, abstract = {In this paper we study how we can design an effective parallel crawler. As the size of the Web grows, it becomes imperative to parallelize a crawling process, in order to finish downloading pages in a reasonable amount of time. We first propose multiple architectures for a parallel crawler and identify fundamental issues related to parallel crawling. Based on this understanding, we then propose metrics to evaluate a parallel crawler, and compare the proposed architectures using 40 million pages ...}, author = {Cho, J. and Molina, Garcia H.}, booktitle = {Proc. of the 11th International World--Wide Web Conference}, citeulike-article-id = {1586892}, citeulike-linkout-0 = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.7355}, keywords = {distributed, web\_crawling}, posted-at = {2007-08-24 00:40:38}, priority = {2}, title = {Parallel crawlers}, url = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.7355}, year = {2002} }

TY - CONF ID - citeulike:1586892 L3 - citeulike-article-id:1586892 N2 - In this paper we study how we can design an effective parallel crawler. As the size of the Web grows, it becomes imperative to parallelize a crawling process, in order to finish downloading pages in a reasonable amount of time. We first propose multiple architectures for a parallel crawler and identify fundamental issues related to parallel crawling. Based on this understanding, we then propose metrics to evaluate a parallel crawler, and compare the proposed architectures using 40 million pages ... TI - Parallel crawlers T2 - Proc. of the 11th International World--Wide Web Conference KW - distributed KW - web_crawling AU - Cho, J AU - Molina, Garcia PY - 2002/// UR - http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.7355 ER -