Discounted Cumulated Gain Based Evaluation of Multiple-Query IR Sessions Export

@incollection{citeulike:2862286, abstract = {IR research has a strong tradition of laboratory evaluation of systems. Such research is based on test collections, pre-defined test topics, and standard evaluation metrics. While recent research has emphasized the user viewpoint by proposing user-based metrics and non-binary relevance assessments, the methods are insufficient for truly user-based evaluation. The common assumption of a single query per topic and session poorly represents real life. On the other hand, one well-known metric for multiple queries per session, instance recall, does not capture early (within session) retrieval of (highly) relevant documents. We propose an extension to the Discounted Cumulated Gain (DCG) metric, the Session-based DCG (sDCG) metric for evaluation scenarios involving multiple query sessions, graded relevance assessments, and open-ended user effort including decisions to stop searching. The sDCG metric discounts relevant results from later queries within a session. We exemplify the sDCG metric with data from an interactive experiment, we discuss how the metric might be applied, and we present research questions for which the metric is helpful.}, author = {J\"{a}rvelin, Kalervo and Price, Susan and Delcambre, Lois and Nielsen, Marianne}, citeulike-article-id = {2862286}, citeulike-linkout-0 = {http://dx.doi.org/10.1007/978-3-540-78646-7_4}, doi = {10.1007/978-3-540-78646-7_4}, journal = {Advances in Information Retrieval}, keywords = {evaluation, ir, ndcg}, pages = {4--15}, posted-at = {2009-11-05 14:30:25}, priority = {4}, title = {Discounted Cumulated Gain Based Evaluation of Multiple-Query IR Sessions}, url = {http://dx.doi.org/10.1007/978-3-540-78646-7_4}, year = {2008} }

TY - CHAP ID - citeulike:2862286 L3 - citeulike-article-id:2862286 N2 - IR research has a strong tradition of laboratory evaluation of systems. Such research is based on test collections, pre-defined test topics, and standard evaluation metrics. While recent research has emphasized the user viewpoint by proposing user-based metrics and non-binary relevance assessments, the methods are insufficient for truly user-based evaluation. The common assumption of a single query per topic and session poorly represents real life. On the other hand, one well-known metric for multiple queries per session, instance recall, does not capture early (within session) retrieval of (highly) relevant documents. We propose an extension to the Discounted Cumulated Gain (DCG) metric, the Session-based DCG (sDCG) metric for evaluation scenarios involving multiple query sessions, graded relevance assessments, and open-ended user effort including decisions to stop searching. The sDCG metric discounts relevant results from later queries within a session. We exemplify the sDCG metric with data from an interactive experiment, we discuss how the metric might be applied, and we present research questions for which the metric is helpful. JF - Advances in Information Retrieval EP - 15 TI - Discounted Cumulated Gain Based Evaluation of Multiple-Query IR Sessions SP - 4 KW - evaluation KW - ir KW - ndcg AU - Järvelin, Kalervo AU - Price, Susan AU - Delcambre, Lois AU - Nielsen, Marianne PY - 2008/// UR - http://dx.doi.org/10.1007/978-3-540-78646-7_4 ER -