Brute Force and Indexed Approaches to Pairwise Document Similarity Comparisons with MapReduce Export

@inproceedings{Lin2009text-mapred, abstract = {This paper explores the problem of computing pairwise similarity on document collections, focusing on the application of \more like this" queries in the life sciences domain. Three MapReduce algorithms are introduced: one based on brute force, a second where the problem is treated as large-scale ad hoc retrieval, and a third based on the Cartesian product of postings lists. Each algorithm supports one or more approximations that trade eectiveness for eciency, the characteristics of which are studied experimentally. Results show that the brute force algorithm is the most ecient of the three when exact similarity is desired. However, the other two algorithms support approximations that yield large ef- ciency gains without signicant loss of eectiveness.}, author = {Lin, Jimmy}, booktitle = {ACM SIGIR}, citeulike-article-id = {5038751}, keywords = {document, index, map, reduce, similarity}, posted-at = {2009-07-02 06:33:26}, priority = {2}, title = {Brute Force and Indexed Approaches to Pairwise Document Similarity Comparisons with MapReduce}, year = {2009} }

TY - CONF ID - Lin2009text-mapred L3 - citeulike-article-id:5038751 N2 - This paper explores the problem of computing pairwise similarity on document collections, focusing on the application of \more like this" queries in the life sciences domain. Three MapReduce algorithms are introduced: one based on brute force, a second where the problem is treated as large-scale ad hoc retrieval, and a third based on the Cartesian product of postings lists. Each algorithm supports one or more approximations that trade eectiveness for eciency, the characteristics of which are studied experimentally. Results show that the brute force algorithm is the most ecient of the three when exact similarity is desired. However, the other two algorithms support approximations that yield large ef- ciency gains without signicant loss of eectiveness. TI - Brute Force and Indexed Approaches to Pairwise Document Similarity Comparisons with MapReduce T2 - ACM SIGIR KW - document KW - index KW - map KW - reduce KW - similarity AU - Lin, Jimmy PY - 2009/// ER -