CiteULike: Tag confidence

Estimating Confidence Using Word Lattices

Thomas Kemp — 2008-06-24T19:21:56-00:00

(1997), pp. 827-830.

For many practical applications of speech recognition systems, it is desirable to have an estimate of confidence for each hypothesized word, i.e. to have an estimate which words of the speech recognizer's output are likely to be correct and which are not reliable. Many of today's speech recognition systems use word lattices as a compact representation of a set of alternative hypothesis. We exploit the use of such word lattices as information sources for the measure-of-confidence tagger JANKA...

Active Learning with Confidence

Mark Dredze — 2008-06-27T21:56:51-00:00

(June 2008), pp. 233-236.

A comparison and combination of methods for OOV word detection and word confidence scoring

TJ Hazen — 2008-06-23T21:08:14-00:00

Acoustics, Speech, and Signal Processing, 2001. Proceedings. (ICASSP '01). 2001 IEEE International Conference on, Vol. 1 (2001), pp. 397-400 vol.1.

This paper examines an approach for combining two different methods for detecting errors in the output of a speech recognizer. The first method attempts to alleviate recognition errors by using an explicit model for detecting the presence of out-of-vocabulary (OOV) words. The second method identifies potentially misrecognized words from a set of confidence features extracted from the recognition process using a confidence scoring model. Since these two methods are inherently different, an approach which combines the techniques can provide significant advantages over either of the individual methods. In experiments in the JUPITER weather domain, we compare and contrast the two approaches and demonstrate the advantage of the combined approach. In comparison to either of the two individual approaches, the combined approach achieves over 25% fewer false acceptances of incorrectly recognized keywords (from 55% to 40%) at a 98% acceptance rate of correctly recognized keywords

Measures of Clade Confidence Do Not Correlate with Accuracy of Phylogenetic Trees

Barry Hall — 2007-04-02T08:10:06-00:00

PLoS Computational Biology, Vol. 3, No. 3. (1 March 2007), e51.

Metrics of phylogenetic tree reliability, such as parametric bootstrap percentages or Bayesian posterior probabilities, represent internal measures of the topological reproducibility of a phylogenetic tree, while the recently introduced aLRT (approximate likelihood ratio test) assesses the likelihood that a branch exists on a maximum-likelihood tree. Although those values are often equated with phylogenetic tree accuracy, they do not necessarily estimate how well a reconstructed phylogeny represents cladistic relationships that actually exist in nature. The authors have therefore attempted to quantify how well bootstrap percentages, posterior probabilities, and aLRT measures reflect the probability that a deduced phylogenetic clade is present in a known phylogeny. The authors simulated the evolution of bacterial genes of varying lengths under biologically realistic conditions, and reconstructed those known phylogenies using both maximum likelihood and Bayesian methods. Then, they measured how frequently clades in the reconstructed trees exhibiting particular bootstrap percentages, aLRT values, or posterior probabilities were found in the true trees. The authors have observed that none of these values correlate with the probability that a given clade is present in the known phylogeny. The major conclusion is that none of the measures provide any information about the likelihood that an individual clade actually exists. It is also found that the mean of all clade support values on a tree closely reflects the average proportion of all clades that have been assigned correctly, and is thus a good representation of the overall accuracy of a phylogenetic tree.

Interactive Information Extraction with Constrained Conditional Random Fields

Trausti Kristjansson — 2007-05-11T22:21:04-00:00

pp. 412-418.

Information Extraction methods can be used to automatically "fill-in" database forms from unstructured data such as Web documents or email. State-of-the-art methods have achieved low error rates but invariably make a number of errors. The goal of an interactive information extraction system is to assist the user in filling in database fields while giving the user confidence in the integrity of the data. The user is presented with an interactive interface that allows both the rapid...

Confidence Estimation for Information Extraction

A Culotta — 2007-05-11T21:57:27-00:00

(2004)

Information extraction techniques automatically create structured databases from unstructured data sources, such as the Web or newswire documents. Despite the successes of these systems, accuracy will always be imperfect.

Predicting the World Cup 2002 in soccer: Performance and confidence of experts and non-experts

Patric Andersson — 2005-06-21T07:38:26-00:00

International Journal of Forecasting, Vol. In Press, Corrected Proof

This paper investigates the forecasting performance and confidence of experts and non-experts. 251 participants with four different levels of knowledge of soccer (ranging between expertise and almost ignorance) took part in a survey and predicted the outcome of the first round of World Cup 2002. The participating experts (i.e., sport journalists, soccer fans, and soccer coaches) and the non-experts were found to be equally accurate and better than chance. A simple prediction rule that followed world rankings outperformed most participants. Experts overestimated their performance and tended to be overconfident, while the opposite tendency was observed for the participants with limited knowledge. Providing non-experts with information did not improve their performance, but increased their confidence.

Intuitive evaluation of likelihood judgment producers: evidence for a confidence heuristic

Paul Price — 2005-05-31T21:52:55-00:00

Journal of Behavioral Decision Making, Vol. 17, No. 1. (16 December 2003), pp. 39-57.

This research tests the hypothesis of Yates et al. (1996) that people prefer judgment producers who make extreme confidence judgments. In each of three experiments, college students evaluated two fictional financial advisors who judged the likelihood that each of several stocks would increase in value. One of the advisors (the moderate advisor) was reasonably well calibrated and the other (the extreme advisor) was overconfident. In all three experiments, participants tended to prefer the extreme advisor. Experiments 2 and 3 showed that the advisors' confidence influenced participants' perception of their knowledge, and Experiment 3 showed that it influenced their perception of the number of categorically correct judgments they made. Both of these variables were, in turn, related to participants' preferences. Experiment 3 also suggested that need for cognition and right-wing authoritarianism are positively related to preference for the extreme advisor. A quantitative model is presented, which captures the basic pattern of results. This model includes the assumption that people use a confidence heuristic; they assume that a more confident advisor makes more categorically correct judgments and is more knowledgeable. Copyright © 2004 John Wiley & Sons, Ltd.

Explicit estimation of visual uncertainty in human motion processing

Erich Graf — 2005-10-10T18:03:07-00:00

Vision Research, Vol. 45, No. 24. (November 2005), pp. 3050-3059.

We examine whether human observers have explicit access to an estimate of their own uncertainty in extrapolating the motion trajectories of moving objects. Objects moved across a display area at constant speed changing direction at short time intervals. Each new direction was obtained by adding a random perturbation to the previous direction. The perturbation distribution was always symmetric with mean zero (no change in direction) but could differ in variability: objects with low directional variability tended to travel in straight lines while objects with high directional variability moved more erratically. Objects eventually disappeared behind the near edge of an occluder. Observers marked a 'capture region' along the far edge of the occluder that they estimated would contain the object when it re-emerged. We varied both occluder width and directional variability across trials and found that observers correctly compensated for these changes. We present a two-stage model of observer performance in which the visual system first estimates the directional variability of the object and then uses this estimate to set a capture region.

The sensory sampling model: theoretical developmentsand empirical findings

Henrik Olsson — 2008-02-03T23:41:44-00:00

Food Quality and Preference, Vol. 11, No. 1-2. (January 2000), pp. 27-34.

The article presents the distinction between Thurstonian and Brunswikian errors in judgment, and provides an informal presentation of the sensory sampling model (SESAM) which is premised on this distinction [Juslin, P., & Olsson, H. (1997). Thurstonian and Brunswikian origins of uncertainty in judgement: a sampling model of confidence in sensory discrimination. Psychological Review, 104, 344-366]. SESAM predicts a disposition towards underestimation of discriminative ability in sensory discrimination tasks not pervasively affected by time pressure or perceptual illusions. This prediction is verified by a review of the literature. A number of recent studies with SESAM are discussed, including the applications to explain the effects of perceptual bias and outcome feedback.

Context, feedback, and the calibration and resolution of confidence in perceptual judgments.

WM Petrusic — 2008-02-03T14:14:07-00:00

Am J Psychol, Vol. 110, No. 4. (1997), pp. 543-572.

The effects of variations in the global task difficulty context on judgmental confidence and confidence calibration were investigated in two experiments requiring perceptual comparisons. In Experiment 1, target judgments of moderate difficulty were embedded in a larger set of more difficult (hard context) or less difficult (easy context) judgments. Decisional response time on the target items was longer in the hard context condition, but there was no effect of difficulty context on target judgment confidence, accuracy, over/underconfidence, calibration, or resolution. In Experiment 2, each subject was exposed to three levels of local judgment difficulty. The global contextual difficulty manipulation involved varying the frequency with which the hard and easy judgments appeared, and the presence or absence of trial-by-trial response feedback was manipulated between subjects. As in Experiment 1, contextual difficulty affected decisional response times but not mean confidence ratings or accuracy. However, we found that providing feedback on a globally difficult task makes calibration worse. Also, resolution (the ability to differentiate correct from incorrect judgments) was found to be superior for easy judgments in a difficult context and for difficult judgments in an easy context. We discuss the implication of these findings for research on confidence and confidence calibration.

Realism of confidence in sensory discrimination.

JV Baranski — 2007-11-30T10:38:19-00:00

Percept Psychophys, Vol. 61, No. 7. (October 1999), pp. 1369-1383.

Is there a common and general basis for confidence in human judgment? Recently, we found that the properties of confidence judgments in the sensory domain mirror those previously established in the cognitive domain; notably, we found underconfidence on easy sensory judgments and overconfidence on hard sensory judgments. In contrast, data from the Uppsala laboratory in Sweden suggest that sensory judgments are unique; they found a pervasive underconfidence bias, with overconfidence being evident only on very hard sensory judgments. Olsson and Winman (1996) attempted to resolve the debate on the basis of methodological issues related to features of the stimulus display in a visual discrimination task. A reanalysis of the data reported in Baranski and Petrusic (1994), together with the findings of a new experiment that controlled stimulus display characteristics, supports the position that the difference between the Canadian and the Swedish data is real and, thus, may reflect cross-national differences in confidence in sensory discrimination.

A model for realism of confidence judgments: implications for underconfidence in sensory discrimination.

WR Ferrell — 2007-11-30T10:35:50-00:00

Percept Psychophys, Vol. 57, No. 2. (February 1995)

In a recent issue of this journal, Björkman, Justlin, and Winman (1993) presented a model of the calibration of subjective confidence judgments for sensory discrimination which they called "subjective distance theory." They proposed that there was a robust underconfidence bias in such judgments, that the model predicted such a bias, and that two different models were needed for the calibration of subjective confidence for cognitive judgments and for sensory ones. This paper addresses issues they raised. It points out that they have not presented a new model, but rather a portion of a more general one, the "decision-variable partition model" originally proposed in Ferrell and McGoey (1980). This paper explores properties of the model and shows, contrary to Björkman, Juslin, and Winman's hypotheses, that the model does not predict underconfidence, that the "hard-easy effect" can be observed with sensory discriminations, and that the model fits not only sensory, but also cognitive judgments.

An application of the poisson race model to confidence calibration.

EC Merkle — 2007-11-30T10:30:57-00:00

J Exp Psychol Gen, Vol. 135, No. 3. (August 2006), pp. 391-408.

In tasks as diverse as stock market predictions and jury deliberations, a person's feelings of confidence in the appropriateness of different choices often impact that person's final choice. The current study examines the mathematical modeling of confidence calibration in a simple dual-choice task. Experiments are motivated by an accumulator model, which proposes that information supporting each alternative accrues on separate counters. The observer responds in favor of whichever alternative's counter first hits a designated threshold. Confidence can then be scaled from the difference between the counters at the time that the observer makes a response. The authors examine the overconfidence result in general and present new findings dealing with the effect of response bias on confidence calibration.

Introduction to this Special Issue on Stochastic and Cognitive Models of Confidence

David Budescu — 2007-11-26T21:18:55-00:00

Journal of Behavioral Decision Making, Vol. 10, No. 3. (1997), pp. 153-155.

No Abstract

The impact of vocabulary preparation on L2 listening comprehension, confidence and strategy use

Ching-Shyang — 2008-03-13T07:02:11-00:00

System, Vol. 35, No. 4. (December 2007), pp. 534-550.

Building on previous studies of the effects of planning on second language learners' (L2) oral narratives and writing, this research reports an investigation of the effects of vocabulary preparation prior to a listening comprehension test on L2 learners' vocabulary performance, listening comprehension, confidence levels and strategy use. The participants were given three different lengths of preparation time to study new vocabulary that would be heard in a listening text. The instruments involved a vocabulary test, a listening comprehension test, and a questionnaire to elicit their confidence levels and strategy use. A semi-structured interview was conducted immediately after the test. The results show that though a consistent pattern was found for the tests of vocabulary and listening comprehension (the more preparation time they had, the higher score they achieved) significant differences between groups were detected only in the vocabulary test but not in the listening comprehension test. In relation to the level of confidence and strategy use, the group with 30-min preparation showed the highest levels of confidence and more strategy use, followed by the group given 1-week preparation. It is concluded that allowing students to study vocabulary before a test could improve their vocabulary knowledge and confidence but not their listening comprehension. In the light of students' responses in the questionnaire and reports in their interviews, the paper discusses a few problems participants had studying the vocabulary and suggestions are made for the teaching of listening.

Does teachers' confidence with CALL equal innovative and integrated use?

G Kessler — 2008-06-04T23:31:52-00:00

Computer Assisted Language Learning, Vol. 21, No. 3. (2008), pp. 269-282.

This paper examines the relationship between confidence and CALL specifically the use of audio and video technology among language teachers. Through logged usage of CALL, the authors tracked seven teachers at two large universities in the United States over a term. These teachers were also interviewed periodically in order to gain insight into their confidence with and use of CALL. Upon data analysis, the authors identified the teachers as less confident, contextually confident and highly confident. Highly confident teachers used technology less often with less integration than the contextually confident teachers. Less confident teachers integrated CALL only in prescribed ways. The authors conclude that CALL teacher preparation may benefit from a focus on developing contextualized confidence within certain teaching domains or types of technology rather than expecting teachers to develop a high level of confidence with technology across domains.

Calculating confidence intervals for prediction error in microarray classification using resampling.

W Jiang — 2008-04-29T20:10:20-00:00

Statistical applications in genetics and molecular biology, Vol. 7 (2008)

Cross-validation based point estimates of prediction accuracy are frequently reported in microarray class prediction problems. However these point estimates can be highly variable, particularly for small sample numbers, and it would be useful to provide confidence intervals of prediction accuracy. We performed an extensive study of existing confidence interval methods and compared their performance in terms of empirical coverage and width. We developed a bootstrap case cross-validation (BCCV) resampling scheme and defined several confidence interval methods using BCCV with and without bias-correction. The widely used approach of basing confidence intervals on an independent binomial assumption of the leave-one-out cross-validation errors results in serious under-coverage of the true prediction error. Two split-sample based methods previously proposed in the literature tend to give overly conservative confidence intervals. Using BCCV resampling, the percentile confidence interval method was also found to be overly conservative without bias-correction, while the bias corrected accelerated (BCa) interval method of Efron returns substantially anti-conservative confidence intervals. We propose a simple bias reduction on the BCCV percentile interval. The method provides mildly conservative inference under all circumstances studied and outperforms the other methods in microarray applications with small to moderate sample sizes.

Improved Boosting Using Confidence-rated Predictions

Robert Schapire — 2005-11-30T08:57:20-00:00

Machine Learning, Vol. 37, No. 3. (1999), pp. 297-336.

. We describe several improvements to Freund and Schapire's AdaBoost boosting algorithm, particularly in a setting in which hypotheses may assign confidences to each of their predictions. We give a simplified analysis of AdaBoost in this setting, and we show how this analysis can be used to find improved parameter settings as well as a refined criterion for training weak hypotheses. We give a specific method for assigning confidences to the predictions of decision trees, a method closely...

Public Confidence in the Management of Radioactive Waste: The Canadian Context: Workshop Proceedings, Ottawa, Canada, 14-18 October 2002

Nea — 2008-05-08T20:19:37-00:00

(27 October 2003)

Trust in a high-concern risk controversy: a comparison of three concepts

George Cvetkovich — 2008-05-13T19:03:39-00:00

Journal of Risk Research, Vol. 10, No. 2. (2007), pp. 223-237.

A community survey examined factors affecting the trust of four groups involved in a high concern controversy over the risks posed by motor boats to the quality of a municipal water supply. In an effort at conceptual integration the survey results were used to examine the relationships between three concepts of trust. Perceived agreement in values between self and four controversy-involved groups was found to be the most powerful predictor of trust of all four groups, as expected by the salient value similarity perspective. Fairness and competency, as expected by the dimensions of trust perspective were also found to be significant predictors of trust. However, judgments of fairness and competency were context specific as indicated by significant correlations with judgments of salient value similarity and self interests. This violates the assumption of universality of the dimensions of trust perspective. Judgments of similarity of values between self and the controversy-involved groups were significantly correlated to self interests. This indicates a conceptual overlap between the salient values similarities perspective and the encapsulated trust perspective.

Perception of risk: the influence of general trust, and general confidence

Michael Siegrist — 2008-05-08T21:16:44-00:00

Journal of Risk Research, Vol. 8, No. 2. (2005), pp. 145-156.

The influence of trust and confidence as personality traits on the perception of various hazards was examined. The distinction between trust and confidence is a key element of certain theories of co-operation, but this dual-mode approach has had little impact on empirical studies. General trust is the belief that other people can be relied on. General confidence is the conviction that everything is under control, and uncertainty is low. It was hypothesized that general trust and general confidence negatively influence risk perception. The hypothesis was tested using data from a random sample of 388 persons living in Switzerland. High levels of trust and confidence reduced perceived risks, compared with low levels of trust and confidence. Age was positively correlated with perceived risk. Gender was a significant predictor for technological hazards, but not for non-technological hazards. Females perceived more risks than males. Results provide strong evidence for the hypothesis that general trust and general confidence have an impact on the perception of new technologies. Practical implications of the results are discussed.

A new look at the psychometric paradigm of perception of hazards

Michael Siegrist — 2005-03-20T16:43:24-00:00

Risk Analysis, Vol. 25, No. 1. (February 2005), pp. 211-222.

The psychometric paradigm has been the most influential model in the field of risk analysis. The "cognitive maps" of hazards produced by the paradigm seem to explain how laypeople perceive the various risks they face. Because most of the studies used aggregated data, analyzed using principal component analysis, it is not known whether the model neglects individual differences in risk perception. There has been much criticism on the fact that few studies have examined individual differences in the cognitive representation of hazards. In order to detect and describe the internal structure of the three-way data, we conducted a three-way component analysis (3MPCA). Data for the present analysis were derived from a mail survey conducted in Switzerland. Participants were asked to judge 9 attributes for 26 hazards. Individual differences in the cognitive representation of hazards were correlated with external variables (e. g., general trust). The results suggest that methods permitting individual differences should be used more frequently and that utilizing different methods could provide greater insight into the cognitive representation of risks.

Confidence measures for spontaneous speech recognition

T Schaaf — 2006-12-01T17:20:13-00:00

Acoustics, Speech, and Signal Processing, 1997. ICASSP-97., 1997 IEEE International Conference on, Vol. 2 (1997), pp. 875-878 vol.2.

For many practical applications of speech recognition systems, it is desirable to have an estimate of confidence for each hypothesized word, i.e. to have an estimate of which words of the output of the speech recognizer are likely to be correct and which are not reliable. We describe the development of the measure of the confidence tagger JANKA, which is able to provide confidence information for the words at the output of the speech recognizer JANUS-3-SR. On a spontaneous German human-to-human database, JANKA achieves a tagging accuracy of 90% at a baseline word accuracy of 82%

Word and phone level acoustic confidence scoring

SO Kamppari — 2006-12-01T17:17:45-00:00

Acoustics, Speech, and Signal Processing, 2000. ICASSP '00. Proceedings. 2000 IEEE International Conference on, Vol. 3 (2000), pp. 1799-1802 vol.3.

This paper presents a word level confidence scoring technique based on a combination of multiple features extracted from the output of a phonetic classifier. The goal of this research was to develop a robust confidence measure based strictly on acoustic information. This research focused on methods for augmenting standard log likelihood ratio techniques with additional information to improve the robustness of the acoustic confidence scores for word recognition tasks. The most successful approach utilized a Fisher linear discriminant projection to reduce a set of acoustic features, extracted from phone level classification results, to a single dimension confidence score. The experiments in this paper were implemented within the JUPITER weather information system. The paper presents results indicating that the technique achieved significant improvements over standard log likelihood ratio techniques for confidence scoring

A Tutorial on Conformal Prediction

Glenn Shafer — 2008-03-25T20:09:52-00:00

Journal of Machine Learning Research, Vol. 9 (March 2008), pp. 371-421.

Conformal prediction uses past experience to determine precise levels of confidence in new predictions. Given an error probability ε, together with a method that makes a prediction y of a label y, it produces a set of labels, typically containing y, that also contains y with probability 1 − ε. Conformal prediction can be applied to any method for producing y: a nearest-neighbor method, a support-vector machine, ridge regression, etc. Conformal prediction is designed for an on-line setting in which labels are predicted successively, each one being revealed before the next is predicted. The most novel and valuable feature of conformal prediction is that if the successive examples are sampled independently from the same distribution, then the successive predictions will be right 1 − ε of the time, even though they are based on an accumulating data set rather than on independent data sets. In addition to the model under which successive examples are sampled independently, other on-line compression models can also use conformal prediction. The widely used Gaussian linear model is one of these. This tutorial presents a self-contained account of the theory of conformal prediction and works through several numerical examples. A more comprehensive treatment of the topic is provided in Algorithmic Learning in a Random World, by Vladimir Vovk, Alex Gammerman, and Glenn Shafer (Springer, 2005).

Distinguishing states of awareness from confidence during retrieval: evidence from amnesia.

S Rajaram — 2006-11-07T18:32:05-00:00

Cogn Affect Behav Neurosci, Vol. 2, No. 3. (September 2002), pp. 227-235.

Two experiments were conducted to determine whether recollective experience is distinguishable from confidence. In Experiment 1, we tested college participants in a within-subjects design and replicated Gardiner and Java's (1990) findings from a between-subjects design. We observed higher remember judgments for words than for nonwords, but higher know judgments for nonwords than for words. For confidence judgments, we obtained greater sure than unsure responses for both words and nonwords. In Experiment 2, we tested amnesic participants and matched control participants. Control participants produced the same pattern of results as college participants, but the results of amnesic participants diverged in an important way. For confidence judgments, the amnesic participants, like the control and college participants, made more sure than unsure judgments to both words and nonwords. But for recollective judgments, amnesic participants did not produce the crossover interaction for words and nonwords. This striking difference between the performance of memory-intact and amnesic participants demonstrates that recollective judgments and confidence that accompany retrieval are not isomorphic psychological experiences.

Error bars in experimental biology

Geoff Cumming — 2007-05-02T18:37:18-00:00

J. Cell Biol., Vol. 177, No. 1. (9 April 2007), pp. 7-11.

Error bars commonly appear in figures in publications, but experimental biologists are often unsure how they should be used and interpreted. In this article we illustrate some basic features of error bars and explain how they can help communicate data and assist correct interpretation. Error bars may show confidence intervals, standard errors, standard deviations, or other quantities. Different types of error bars give quite different information, and so figure legends must make clear what error bars represent. We suggest eight simple rules to assist with effective use and interpretation of error bars. 10.1083/jcb.200611141

Communication in computer science classrooms: understanding defensive climates as a means of creating supportive behaviors

Kathy Garvin-Doxas — 2005-09-15T23:42:32-00:00

J. Educ. Resour. Comput., Vol. 4, No. 1. (March 2004)

Soft Syntactic Constraints for Hierarchical Phrased-Based Translation

Yuval Marton — 2008-06-27T22:03:29-00:00

(June 2008), pp. 1003-1011.

Computing Confidence Scores for All Sub Parse Trees

Feng Lin — 2008-06-27T21:53:25-00:00

(June 2008), pp. 217-220.

Gaining confidence in high-throughput protein interaction networks.

JS Bader — 2005-06-14T02:53:11-00:00

Nat Biotechnol, Vol. 22, No. 1. (January 2004), pp. 78-85.

Although genome-scale technologies have benefited from statistical measures of data quality, extracting biologically relevant pathways from high-throughput proteomics data remains a challenge. Here we develop a quantitative method for evaluating proteomics data. We present a logistic regression approach that uses statistical and topological descriptors to predict the biological relevance of protein-protein interactions obtained from high-throughput screens for yeast. Other sources of information, including mRNA expression, genetic interactions and database annotations, are subsequently used to validate the model predictions without bias or cross-pollution. Novel topological statistics show hierarchical organization of the network of high-confidence interactions: protein complex interactions extend one to two links, and genetic interactions represent an even finer scale of organization. Knowledge of the maximum number of links that indicates a significant correlation between protein pairs (correlation distance) enables the integrated analysis of proteomics data with data from genetics and gene expression. The type of analysis presented will be essential for analyzing the growing amount of genomic and proteomics data in model organisms and humans.

A direct comparison of protein interaction confidence assignment schemes

Silpa Suthram — 2006-07-26T21:44:25-00:00

BMC Bioinformatics, Vol. 7 (26 July 2006), 360.

Qualified predictions for microarray and proteomics pattern diagnostics with confidence machines.

T Bellotti — 2006-08-22T14:17:11-00:00

Int J Neural Syst, Vol. 15, No. 4. (August 2005), pp. 247-258.

We focus on the problem of prediction with confidence and describe a recently developed learning algorithm called transductive confidence machine for making qualified region predictions. Its main advantage, in comparison with other classifiers, is that it is well-calibrated, with number of prediction errors strictly controlled by a given predefined confidence level. We apply the transductive confidence machine to the problems of acute leukaemia and ovarian cancer prediction using microarray and proteomics pattern diagnostics, respectively. We demonstrate that the algorithm performs well, yielding well-calibrated and informative predictions whilst maintaining a high level of accuracy.

Statistics with Confidence

2008-07-08T10:57:00-00:00

This highly popular introduction to confidence intervals has been thoroughly updated and expanded. It includes methods for using confidence intervals, with illustrative worked examples and extensive guidelines and checklists to help the novice.

I Can't Believe I'm Stupid

Andy Egan — 2007-10-24T17:18:12-00:00

Philosophical Perspectives, Vol. 19, No. 1. (2005), pp. 77-93.

Sifting the evidence---what's wrong with significance tests? Another comment on the role of statistical methods

Jonathan Sterne — 2005-08-31T20:01:33-00:00

BMJ, Vol. 322, No. 7280. (27 January 2001), pp. 226-231.

Estimating the Variance in Nonparametric Regression-What is a Reasonable Choice?

Holger Dette — 2006-07-18T22:39:43-00:00

The exact mean-squared error (MSE) of estimators of the variance in nonparametric regression based on quadratic forms is investigated. In particular, two classes of estimators are compared: Hall, Kay and Titterington's optimal difference-based estimators and a class of ordinary difference-based estimators which generalize methods proposed by Rice and Gasser, Sroka and Jennen-Steinmetz. For small sample sizes the MSE of the first estimator is essentially increased by the magnitude of the integrated first two squared derivatives of the regression function. It is shown that in many situations ordinary difference-based estimators are more appropriate for estimating the variance, because they control the bias much better and hence have a much better overall performance. It is also demonstrated that Rice's estimator does not always behave well. Data-driven guidelines are given to select the estimator with the smallest MSE.

Calibration-Based Predictive Distributions: An Application of Prequential Analysis to Interest Rates, Money, Prices, and Output

John Kling — 2008-05-26T17:07:57-00:00

The Journal of Business, Vol. 62, No. 4. (1989), pp. 477-499.

Some techniques of probability forecasting are applied to time-series data on interest rates, money stock, consumer prices, and output. A sequential method for debiasing (recalibrating) predictive distributions based on previously issued distributions and outcomes is developed, and our estimated sequences of unadjusted and recalibrated distributions are tested for calibration. After recalibration, the calibration hypothesis cannot be rejected for most of the time-series and forecast horizons. Furthermore, traditional point forecasts can be improved when the forecast are derived from recalibrated distributions.

Towards improving trust in context-aware systems by displaying system confidence

Stavros Antifakos — 2008-01-22T20:12:44-00:00

(2005), pp. 9-14.

A confidence-based framework for disambiguating geographic terms

E Rauch — 2006-10-29T18:48:45-00:00

(2003)

We describe a purely confidence-based geographic term disambiguation system that crucially relies on the notion of "positive" and "negative" context and methods for combining confidence-based disambiguation with measures of relevance to a user's query.