CiteULike: Tag judgement

Why Most Published Research Findings Are False

John Ioannidis — 2007-03-09T15:04:00-00:00

PLoS Medicine, Vol. 2, No. 8. (1 August 2005), e124.

Summary There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

Scientific Judgment and the Limits of Conflict-of-Interest Policies

Kevin Elliott — 2008-01-07T14:28:15-00:00

Accountability in Research, Vol. 15, No. 1. (2008), pp. 1-29.

This article argues that the three major elements of typical university conflict-of-interest (COI) policies (i.e., disclosure, management, and elimination of conflicts via divestiture or recusal) are likely to be insufficient for screening out many worrisome influences of financial COIs. Current psychological research challenges the effectiveness of disclosure, management plans are unlikely to address the wide range of ways that financial COIs can influence scientific judgment, and it is often impractical to eliminate conflicts. Identifying the limits of these policies highlights the importance of considering alternative strategies, such as encouraging more independently funded research, in order to maintain the integrity of science.

Acceptable regret in medical decision making.

B Djulbegovic — 2007-03-10T11:07:32-00:00

Med Hypotheses, Vol. 53, No. 3. (September 1999), pp. 253-259.

When faced with medical decisions involving uncertain outcomes, the principles of decision theory hold that we should select the option with the highest expected utility to maximize health over time. Whether a decision proves right or wrong can be learned only in retrospect, when it may become apparent that another course of action would have been preferable. This realization may bring a sense of loss, or regret. When anticipated regret is compelling, a decision maker may choose to violate expected utility theory to avoid regret. We formulate a concept of acceptable regret in medical decision making that explicitly introduces the patient's attitude toward loss of health due to a mistaken decision into decision making. In most cases, minimizing expected regret results in the same decision as maximizing expected utility. However, when acceptable regret is taken into consideration, the threshold probability below which we can comfortably withhold treatment is a function only of the net benefit of the treatment, and the threshold probability above which we can comfortably administer the treatment depends only on the magnitude of the risks associated with the therapy. By considering acceptable regret, we develop new conceptual relations that can help decide whether treatment should be withheld or administered, especially when the diagnosis is uncertain. This may be particularly beneficial in deciding what constitutes futile medical care.

The role of judgement

Michael Luntley — 2005-09-21T12:29:46-00:00

Philosophical Explorations, Vol. 8, No. 3. (September 2005), pp. 281-295.

Publication Decisions and Their Possible Effects on Inferences Drawn from Tests of Significance--Or Vice Versa

Theodore Sterling — 2007-02-07T11:56:06-00:00

Journal of the American Statistical Association, Vol. 54, No. 285. (1959), pp. 30-34.

There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs-an "error of the first kind"-and is published. Significant results published in these fields are seldom verified by independent replication. The possibility thus arises that the literature of such a field consists in substantial part of false conclusions resulting from errors of the first kind in statistical tests of significance.

Reactions to Uncertainty and the Accuracy of Diagnostic Mammography

Patricia Carney — 2007-01-23T12:08:50-00:00

Journal of General Internal Medicine

A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction.

EM Antman — 2007-03-10T13:56:34-00:00

JAMA, Vol. 268, No. 2. (8 July 1992), pp. 240-248.

OBJECTIVE--To examine the temporal relationship between accumulating data from randomized control trials of treatments for myocardial infarction and the recommendations of clinical experts writing review articles and textbook chapters. DATA SOURCES--(1) MEDLINE search from 1966 to present; search terms used were myocardial infarction, clinical trials, multicenter studies, double-blind method, meta-analysis, and the text word "random:"; (2) references from pertinent articles and books; and (3) all editions of English-language general medical texts and manuals and review articles on treatment of myocardial infarction. STUDY SELECTION--Randomized control trials of therapies for reducing the risk of total mortality in myocardial infarction (acute and secondary prevention). Review articles and textbook chapters dealing with the general clinical management of patients with myocardial infarction. DATA EXTRACTION--Two authors read the material and recorded the results; disagreements were resolved by conference. DATA SYNTHESIS--We used the technique of cumulative meta-analysis (performing a new meta-analysis when the results of a new clinical trial are published) and compared the results with the recommendations of the experts for various treatments for myocardial infarction. Discrepancies were detected between the meta-analytic patterns of effectiveness in the randomized trials and the recommendations of reviewers. Review articles often failed to mention important advances or exhibited delays in recommending effective preventive measures. In some cases, treatments that have no effect on mortality or are potentially harmful continued to be recommended by several clinical experts. CONCLUSIONS--Finding and analyzing all therapeutic trials in a given field has become such a difficult and specialized task that the clinical experts called on to summarize the evidence in a timely fashion need access to better databases and new statistical techniques to assist them in this important task.

Evidence for practice, epistemology, and critical reflection

Avis — 2006-09-12T07:17:55-00:00

Nursing Philosophy, Vol. 7, No. 4. (October 2006), pp. 216-224.

Natural Reasons: Personality and Polity

SL Hurley — 2007-03-31T13:14:48-00:00

(01 August 1992)

Hurley here revives a classical idea about rationality in a modern framework, by developing analogies between the structure of personality and the structure of society in the context of contemporary work in philosophy of mind, ethics, decision theory and social choice theory. The book examines the rationality of decisions and actions, and illustrates the continuity of philosophy of mind on the one hand, and ethics and jurisprudence on the other. A major thesis of the book is that arguments drawn from the philosophy of mind may be used to undermine widely-held subjectivist positions in ethics and politico-economic theory. The work is inspired by the philosophies of Wittgenstein and Davidson, but goes on to connect their arguments about interpretation with formal work in decision theory and social choice theory, and with the theory of adjudication.

Prediction, Explanation, and Dioxin Biochemistry: Science in Public Policy

Heather Douglas — 2006-11-09T13:42:33-00:00

Foundations of Chemistry, Vol. V6, No. 1. (1 January 2004), pp. 49-63.

Rationality and the Emotions

Jon Elster — 2007-11-08T19:27:24-00:00

The Economic Journal, Vol. 106, No. 438. (1996), pp. 1386-1397.

Practical Reasoning and Weakness of the Will

Michael Bratman — 2007-11-27T20:33:50-00:00

Noûs, Vol. 13, No. 2. (1979), pp. 153-171.

Anomalies: Intertemporal Choice

George Loewenstein — 2007-09-24T21:19:41-00:00

The Journal of Economic Perspectives, Vol. 3, No. 4. (1989), pp. 181-193.

Plato, Hare and Davidson on Akrasia

CCW Taylor — 2007-11-27T20:29:14-00:00

Acting for Reasons

Robert Audi — 2007-10-25T00:07:12-00:00

The Philosophical Review, Vol. 95, No. 4. (1986), pp. 511-546.

Motivation: Essentially Motivation-Constituting Attitudes

Alfred Mele — 2007-11-08T20:22:42-00:00

The Philosophical Review, Vol. 104, No. 3. (1995), pp. 387-423.

Desiring the Bad: An Essay in Moral Psychology

Michael Stocker — 2007-11-27T20:24:15-00:00

The Journal of Philosophy, Vol. 76, No. 12. (1979), pp. 738-753.

Weakness of Will and Practical Judgment

Robert Audi — 2007-10-24T23:58:41-00:00

Noûs, Vol. 13, No. 2. (1979), pp. 173-196.

Is Akratic Action Unfree?

Alfred Mele — 2007-10-24T23:55:23-00:00

Philosophy and Phenomenological Research, Vol. 46, No. 4. (1986), pp. 673-679.

Rationality, Morality, and Collective Action

Jon Elster — 2007-11-08T20:19:25-00:00

Ethics, Vol. 96, No. 1. (1985), pp. 136-155.

Akrasia, Self-Control, and Second-Order Desires

Alfred Mele — 2007-11-28T18:03:16-00:00

Noûs, Vol. 26, No. 3. (1992), pp. 281-302.

Jackson on Weakness of Will

Christopher Cordner — 2007-11-28T18:00:57-00:00

Recent Work on Addiction and Responsible Agency

Gideon Yaffe — 2007-11-08T19:22:50-00:00

Philosophy and Public Affairs, Vol. 30, No. 2. (2001), pp. 178-221.

The Judgment of a Weak Will

Sergio Tenenbaum — 2007-10-24T23:42:35-00:00

Philosophy and Phenomenological Research, Vol. 59, No. 4. (1999), pp. 875-911.

In trying to explain the possibility of akrasia (weakness of will), it seems plausible to deny that there is a conceptual connection between motivation (what one wants) and evaluation (what one judges to be good); akrasia occurs when the agent is (most) motivated to do something that she does not judge to be good (all things considered). However, it is hard to see how such accounts could respect our intuition that the akratic agent acts freely, or that there is a difference between akrasia and compulsion. It is also hard to see how such accounts could be extended to the realm of theoretical reason, but this is generally not taken to be a problem, because it is generally assumed that there is no similar phenomenon in the realm of theoretical reason. This paper argues that there is such a thing as theoretical akrasia, and that we can find a characterization of this phenomenon in Descartes's Meditations. Drawing on certain passages in the Meditations, we can construct an account of theoretical akrasia; this account can then be adapted to resolve the original problem of akrasia in the realm of practical reason. The account asserts that there is a conceptual connection between motivation and evaluation in free action; it also enables us to show how the akratic agent is still acting freely when he does something that he does not judge to be the best all things considered.

Explaining Action

Kieran Setiya — 2007-11-08T20:42:29-00:00

The Philosophical Review, Vol. 112, No. 3. (2003), pp. 339-393.

Time Discounting and Time Preference: A Critical Review

Shane Frederick — 2007-10-28T20:56:29-00:00

Journal of Economic Literature, Vol. 40, No. 2. (2002), pp. 351-401.

The Social and Political Sources of Akrasia

Amélie Rorty — 2007-11-08T20:37:28-00:00

Ethics, Vol. 107, No. 4. (1997), pp. 644-657.

The Seductive Allure of Neuroscience Explanations

Deena Weisberg — 2008-02-16T10:05:06-00:00

J. Cogn. Neurosci., Vol. 20, No. 3. (1 March 2008), pp. 470-477.

Explanations of psychological phenomena seem to generate more public interest when they contain neuroscientific information. Even irrelevant neuroscience information in an explanation of a psychological phenomenon may interfere with people's abilities to critically consider the underlying logic of this explanation. We tested this hypothesis by giving naive adults, students in a neuroscience course, and neuroscience experts brief descriptions of psychological phenomena followed by one of four types of explanation, according to a 2 (good explanation vs. bad explanation) x 2 (without neuroscience vs. with neuroscience) design. Crucially, the neuroscience information was irrelevant to the logic of the explanation, as confirmed by the expert subjects. Subjects in all three groups judged good explanations as more satisfying than bad ones. But subjects in the two nonexpert groups additionally judged that explanations with logically irrelevant neuroscience information were more satisfying than explanations without. The neuroscience information had a particularly striking effect on nonexperts' judgments of bad explanations, masking otherwise salient problems in these explanations.

Losing sight of oneself in the above-average effect: When egocentrism, focalism, and group diffuseness collide

Zlatan Krizan — 2008-08-05T13:17:57-00:00

Journal of Experimental Social Psychology, Vol. 44, No. 4. (July 2008), pp. 929-942.

Four experiments examined the relative influence of three causal processes in the above-average effect (AAE) and related comparative biases: (a) egocentrism, (b) focalism, and (c) referent group diffuseness. By manipulating the inclusion or exclusion of the self from the referent group (Experiments 1-3) or target group (Experiment 4), the relative contributions of each influence were assessed. In direct comparisons, single peers were systematically judged more favorably relative to groups including the self, suggesting that egocentrism plays a lesser role than focalism or group diffuseness. Thus, in response to the question such as "How friendly is Nancy compared to the rest of us?" the answer tends to be "More friendly."

The Cambridge Handbook of Thinking and Reasoning

Keith Holyoak — 2005-08-02T09:26:20-00:00

(18 April 2005)

Written by foremost authorities from cognitive psychology, cognitive science, and cognitive neuroscience, the chapters of this reference summarize basic concepts and facts of a major topic, sketch its history, and analyze the progress its research is currently making. The volume also includes work related to developmental, social and clinical psychology, philosophy, economics, artificial intelligence, linguistics, education, law, and medicine. The Cambridge Handbook of Thinking and Reasoning comprises the first comprehensive and authoritative handbook for all core topics within the fields of thinking and reasoning. The Cambridge Handbook of Thinking and Reasoning is the first comprehensive and authoritative handbook covering all the core topics of the field of thinking and reasoning. Written by the foremost experts from cognitive psychology, cognitive science, and cognitive neuroscience, individual chapters summarize basic concepts and findings for a major topic, sketch its history, and give a sense of the directions in which research is currently heading. The volume also includes work related to developmental, social and clinical psychology, philosophy, economics, artificial intelligence, linguistics, education, law, and medicine. Scholars and students in all these fields and others will find this to be a valuable collection.

On Making the Right Choice: The Deliberation-Without-Attention Effect

Ap Dijksterhuis — 2006-02-17T17:51:40-00:00

Science, Vol. 311, No. 5763. (17 February 2006), pp. 1005-1007.

Contrary to conventional wisdom, it is not always advantageous to engage in thorough conscious deliberation before choosing. On the basis of recent insights into the characteristics of conscious and unconscious thought, we tested the hypothesis that simple choices (such as between different towels or different sets of oven mitts) indeed produce better results after conscious thought, but that choices in complex matters (such as between different houses or different cars) should be left to unconscious thought. Named the "deliberation-without-attention" hypothesis, it was confirmed in four studies on consumer choice, both in the laboratory as well as among actual shoppers, that purchases of complex products were viewed more favorably when decisions had been made in the absence of attentive deliberation.

Expert Political Judgment : How Good is It? How Can We Know?

Philip Tetlock — 2006-03-18T01:21:28-00:00

(05 July 2005)

The intelligence failures surrounding the invasion of Iraq dramatically illustrate the necessity of developing standards for evaluating expert opinion. This book fills that need. Here, Philip E. Tetlock explores what constitutes good judgment in predicting future events, and looks at why experts are often wrong in their forecasts. Tetlock first discusses arguments about whether the world is too complex for people to find the tools to understand political phenomena, let alone predict the future. He evaluates predictions from experts in different fields, comparing them to predictions by well-informed laity or those based on simple extrapolation from current trends. He goes on to analyze which styles of thinking are more successful in forecasting. Classifying thinking styles using Isaiah Berlin's prototypes of the fox and the hedgehog, Tetlock contends that the fox--the thinker who knows many little things, draws from an eclectic array of traditions, and is better able to improvise in response to changing events--is more successful in predicting the future than the hedgehog, who knows one big thing, toils devotedly within one tradition, and imposes formulaic solutions on ill-defined problems. He notes a perversely inverse relationship between the best scientific indicators of good judgement and the qualities that the media most prizes in pundits--the single-minded determination required to prevail in ideological combat. Clearly written and impeccably researched, the book fills a huge void in the literature on evaluating expert opinion. It will appeal across many academic disciplines as well as to corporations seeking to develop standards for judging expert decision-making.