Is relevance relevant? User relevance ratings may not predict the impact of Internet search on decision outcomes.

Enrico W Coiera — 2008-04-28T07:26:13-00:00

Journal of the American Medical Informatics Association : JAMIA (24 April 2008)

OBJECTIVE A common measure of Internet search engine effectiveness is its ability to find documents that a user perceives as 'relevant'. This study sought to test whether user provided relevance ratings for documents retrieved by an Internet search engine correlate with the decision outcome after use of a search engine. DESIGN 227 university students were asked to answer four randomly assigned consumer health questions, then to conduct an Internet search on one of two randomly assigned search engines of different performance, and to again answer the question. MEASUREMENTS PARTICIPANTS were asked to provide a relevance score for each document retrieved as well as a pre and post search answer to each question. RESULTS User relevance rankings had little or no predictive power. Relevance rankings were unable to predict whether the user of a search engine could correctly answer a question after search and could not differentiate between two search engines with statistically different performance in the hands of users. Only when users had strong prior knowledge of the questions, and the decision task was of low complexity, did relevance appear to have modest predictive power. CONCLUSION User provided relevance rankings taken in isolation seem to be of limited to no value when designing a search engine that will be used in a general-purpose setting. Relevance rankings may have a place in situations in which experts provide rankings, and decision tasks are of complexity commensurate with the abilities of the raters. A more natural metric of search engine performance may be a user's ability to accurately complete a task, as this removes the inherent subjectivity of relevance rankings, and provides a direct and repeatable outcome measure which directly correlates with the performance of the search technology in the hands of users.

Long-Term Patterns of Online Evidence Retrieval Use in General Practice: A 12-Month Study

Farah Magrabi — 2008-03-20T08:27:13-00:00

J Med Internet Res, Vol. 10, No. 1. (Mar 2008)

Background: Provision of online evidence at the point of care is one strategy that could provide clinicians with easy access to up-to-date evidence in clinical settings in order to support evidence-based decision making. Objective: The aim was to determine long-term use of an online evidence system in routine clinical practice. Methods: This was a prospective cohort study. 59 clinicians who had a computer with Internet access in their consulting room participated in a 12-month trial of Quick Clinical, an online evidence system specifically designed around the needs of general practitioners (GPs). Patterns of use were determined by examination of computer logs and survey analysis. Results: On average, 9.9 searches were conducted by each GP in the first 2 months of the study. After this, usage dropped to 4.4 searches per GP in the third month and then levelled off to between 0.4 and 2.6 searches per GP per month. The majority of searches (79.2%, 2013/2543) were conducted during practice hours (between 9 am and 5 pm) and on weekdays (90.7%, 2315/2543). The most frequent searches related to diagnosis (33.6%, 821/2291) and treatment (34.5%, 844/2291). Conclusion: GPs will use an online evidence retrieval system in routine practice; however, usage rates drop significantly after initial introduction of the system. Long-term studies are required to determine the extent to which GPs will integrate the use of such technologies into their everyday clinical practice and how this will affect the satisfaction and health outcomes of their patients.

CiteULike: Gaetan's Coiera

Is relevance relevant? User relevance ratings may not predict the impact of Internet search on decision outcomes.

Long-Term Patterns of Online Evidence Retrieval Use in General Practice: A 12-Month Study