CiteULike: acslab's confidence-judgment

The Effects of Averaging Subjective Probability Estimates Between and Within Judges

Dan Ariely — 2008-04-24T21:40:37-00:00

Journal of Experimental Psychology: Applied, Vol. 6, No. 2. (1 June 2000), pp. 130-147.

The average probability estimate of J > 1 judges is generally better than its components. Two studies test 3 predictions regarding averaging that follow from theorems based on a cognitive model of the judges and idealizations of the judgment situation. Prediction 1 is that the average of conditionally pairwise independent estimates will be highly diagnostic, and Prediction 2 is that the average of dependent estimates (differing only by independent error terms) may be well calibrated. Prediction 3 contrasts between- and within-subject averaging. Results demonstrate the predictions' robustness by showing the extent to which they hold as the information conditions depart from the ideal and as J increases. Practical consequences are that (a) substantial improvement can be obtained with as few as 2–6 judges and (b) the decision maker can estimate the nature of the expected improvement by considering the information conditions.

Confidence in aggregation of expert opinions

David Budescu — 2008-04-24T21:35:56-00:00

Acta Psychologica, Vol. 104, No. 3. (June 2000), pp. 371-398.

We investigate the case of a single decision maker (DM) who obtains probabilistic forecasts regarding the occurrence of a unique target event from J distinct, symmetric, and equally diagnostic expert advisors (judges). The paper begins with a mathematical model of DM's aggregation process of expert opinions, in which confidence in the final aggregate is shown to be inversely related to its perceived variance. As such, confidence is expected to vary as a function of factors such as the number of experts, the total number of cues, the fraction of cues available to each expert, the level of inter-expert overlap in information, and the range of experts' opinions. In the second part of the paper, we present results from two experiments that support the main (ordinal) predictions of the model.

The effects of asymmetry among advisors on the aggregation of their opinions

David Budescu — 2008-04-24T21:31:01-00:00

Organizational Behavior and Human Decision Processes, Vol. 90, No. 1. (January 2003), pp. 178-194.

We investigate the case of a Decision Maker (DM) who obtains probabilistic forecasts regarding the occurrence of a target event from J distinct, asymmetric advisors. In this context, asymmetry is induced by manipulating: (1) amount of information (number of diagnostic cues) available to each advisor and (2) quality (accuracy) of advisors' previous forecasts. Empirical results from two experiments indicate that the DM's final estimate can be described as a weighted average of advisor forecasts, where the weights are sensitive to both sources of asymmetry. This work extends the model derived by Budescu and Rantilla (2000) for the DMs confidence in the aggregate to accommodate advisor asymmetry. As in the symmetric case, the DM's confidence in the weighted average of the forecasts is a function of the number of judges, the total number of cues, the (inferred) inter-judge correlation, and the level of inter-judge overlap in information. The extended model predicts that confidence increases as a function of asymmetry among judges. Empirical results support the main (ordinal) predictions of the model, including the predicted effect of inter-judge asymmetry.