CiteULike: nelmor's decision

Prefrontal Coding of Temporally Discounted Values during Intertemporal Choice

Soyoun Kim — 2008-07-11T08:28:44-00:00

Neuron, Vol. 59, No. 1. (10 July 2008), pp. 161-172.

Summary Reward from a particular action is seldom immediate, and the influence of such delayed outcome on choice decreases with delay. It has been postulated that when faced with immediate and delayed rewards, decision makers choose the option with maximum temporally discounted value. We examined the preference of monkeys for delayed reward in an intertemporal choice task and the neural basis for real-time computation of temporally discounted values in the dorsolateral prefrontal cortex. During this task, the locations of the targets associated with small or large rewards and their corresponding delays were randomly varied. We found that prefrontal neurons often encoded the temporally discounted value of reward expected from a particular option. Furthermore, activity tended to increase when discounted values for targets were presented in the neuron's preferred direction, suggesting that activity related to temporally discounted values in the prefrontal cortex might determine the animal's behavior during intertemporal choice.

Perceptual accuracy and conflicting effects of certainty on risk-taking behaviour

Sharoni Shafir — 2008-06-12T04:36:15-00:00

Nature, Vol. 453, No. 7197., pp. 917-920.

Dissociating the Role of the Orbitofrontal Cortex and the Striatum in the Computation of Goal Values and Prediction Errors

Todd Hare — 2008-05-29T14:39:09-00:00

J. Neurosci., Vol. 28, No. 22. (28 May 2008), pp. 5623-5630.

To make sound economic decisions, the brain needs to compute several different value-related signals. These include goal values that measure the predicted reward that results from the outcome generated by each of the actions under consideration, decision values that measure the net value of taking the different actions, and prediction errors that measure deviations from individuals' previous reward expectations. We used functional magnetic resonance imaging and a novel decision-making paradigm to dissociate the neural basis of these three computations. Our results show that they are supported by different neural substrates: goal values are correlated with activity in the medial orbitofrontal cortex, decision values are correlated with activity in the central orbitofrontal cortex, and prediction errors are correlated with activity in the ventral striatum. 10.1523/JNEUROSCI.1309-08.2008

Value Representations in the Primate Striatum during Matching Behavior

Brian Lau — 2008-05-08T09:40:23-00:00

Neuron, Vol. 58, No. 3. (8 May 2008), pp. 451-463.

Summary Choosing the most valuable course of action requires knowing the outcomes associated with the available alternatives. The striatum may be important for representing the values of actions. We examined this in monkeys performing an oculomotor choice task. The activity of phasically active neurons (PANs) in the striatum covaried with two classes of information: action-values and chosen-values. Action-value PANs were correlated with value estimates for one of the available actions, and these signals were frequently observed before movement execution. Chosen-value PANs were correlated with the value of the action that had been chosen, and these signals were primarily observed later in the task, immediately before or persistently after movement execution. These populations may serve distinct functions mediated by the striatum: some PANs may participate in choice by encoding the values of the available actions, while other PANs may participate in evaluative updating by encoding the reward value of chosen actions.

Smokers' brains compute, but ignore, a fictive error signal in a sequential investment task

Pearl Chiu — 2008-03-27T04:34:10-00:00

Nature Neuroscience, Vol. 11, No. 4. (02 March 2008), pp. 514-520.

Neural correlates of perceptual learning in a sensory-motor, but not a sensory, cortical area.

Chi-Tat Law — 2008-03-17T17:04:07-00:00

Nat Neurosci (9 March 2008)

This study aimed to identify neural mechanisms that underlie perceptual learning in a visual-discrimination task. We trained two monkeys (Macaca mulatta) to determine the direction of visual motion while we recorded from their middle temporal area (MT), which in trained monkeys represents motion information that is used to solve the task, and lateral intraparietal area (LIP), which represents the transformation of motion information into a saccadic choice. During training, improved behavioral sensitivity to weak motion signals was accompanied by changes in motion-driven responses of neurons in LIP, but not in MT. The time course and magnitude of the changes in LIP correlated with the changes in behavioral sensitivity throughout training. Thus, for this task, perceptual learning does not appear to involve improvements in how sensory information is represented in the brain, but rather how the sensory representation is interpreted to form the decision that guides behavior.

Integrating hippocampus and striatum in decision-making.

Adam Johnson — 2008-03-17T15:13:24-00:00

Curr Opin Neurobiol (27 February 2008)

Learning and memory and navigation literatures emphasize interactions between multiple memory systems: a flexible, planning-based system and a rigid, cached-value system. This has profound implications for decision-making. Recent conceptualizations of flexible decision-making employ prospection and projection arising from a network involving the hippocampus. Recent recordings from rodent hippocampus in decision-making situations have found transient forward-shifted representations. Evaluation of that prediction and subsequent action-selection probably occurs downstream (e.g. in orbitofrontal cortex, in ventral and dorsomedial striatum). Classically, striatum has been identified as a crucial component of the less-flexible, incremental system. Current evidence, however, suggests that striatum is involved in both flexible and stimulus-response decision-making, with dorsolateral striatum involved in stimulus-response strategies and ventral and dorsomedial striatum involved in goal-directed strategies.

Orbitofrontal cortex, decision-making and drug addiction

G Schoenbaum — 2006-01-26T22:09:38-00:00

Trends in Neurosciences, Vol. 29, No. 2. (February 2006), pp. 116-124.

The orbitofrontal cortex, as a part of prefrontal cortex, is implicated in executive function. However, within this broad region, the orbitofrontal cortex is distinguished by its unique pattern of connections with crucial subcortical associative learning nodes, such as basolateral amygdala and nucleus accumbens. By virtue of these connections, the orbitofrontal cortex is uniquely positioned to use associative information to project into the future, and to use the value of perceived or expected outcomes to guide decisions. This review will discuss recent evidence that supports this proposal and will examine evidence that loss of this signal, as the result of drug-induced changes in these brain circuits, might account for the maladaptive decision-making that characterizes drug addiction.

Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards

Matthew Roesch — 2007-11-27T17:36:53-00:00

Nature Neuroscience, Vol. 10, No. 12. (18 November 2007), pp. 1615-1624.

Hold Your Horses: Impulsivity, Deep Brain Stimulation, and Medication in Parkinsonism

Michael Frank — 2007-11-20T16:59:08-00:00

Science (25 October 2007), 1146157.

Deep brain stimulation (DBS) of the subthalamic nucleus dramatically improves the motor symptoms of Parkinson's disease, but causes cognitive side effects such as impulsivity. Here we show that DBS selectively interferes with the normal ability to slow down when faced with decision conflict. While on DBS, patients actually sped up under high conflict conditions. This form of impulsivity was not affected by dopaminergic medication status. Instead, medication impaired patients' ability to learn from negative decision outcomes. These findings implicate independent mechanisms leading to impulsivity in treated Parkinson's patients, and were predicted by a single neurocomputational model of the basal ganglia. 10.1126/science.1146157

The neural correlates of subjective value during intertemporal choice.

Joseph W Kable — 2007-11-14T23:00:04-00:00

Nat Neurosci (4 November 2007)

Neuroimaging studies of decision-making have generally related neural activity to objective measures (such as reward magnitude, probability or delay), despite choice preferences being subjective. However, economic theories posit that decision-makers behave as though different options have different subjective values. Here we use functional magnetic resonance imaging to show that neural activity in several brain regions-particularly the ventral striatum, medial prefrontal cortex and posterior cingulate cortex-tracks the revealed subjective value of delayed monetary rewards. This similarity provides unambiguous evidence that the subjective value of potential rewards is explicitly represented in the human brain.

Reinforcement Learning Signals in the Human Striatum Distinguish Learners from Nonlearners during Reward-Based Decision Making

Tom Schonberg — 2007-11-22T11:18:55-00:00

J. Neurosci., Vol. 27, No. 47. (21 November 2007), pp. 12860-12867.

The computational framework of reinforcement learning has been used to forward our understanding of the neural mechanisms underlying reward learning and decision-making behavior. It is known that humans vary widely in their performance in decision-making tasks. Here, we used a simple four-armed bandit task in which subjects are almost evenly split into two groups on the basis of their performance: those who do learn to favor choice of the optimal action and those who do not. Using models of reinforcement learning we sought to determine the neural basis of these intrinsic differences in performance by scanning both groups with functional magnetic resonance imaging. We scanned 29 subjects while they performed the reward-based decision-making task. Our results suggest that these two groups differ markedly in the degree to which reinforcement learning signals in the striatum are engaged during task performance. While the learners showed robust prediction error signals in both the ventral and dorsal striatum during learning, the nonlearner group showed a marked absence of such signals. Moreover, the magnitude of prediction error signals in a region of dorsal striatum correlated significantly with a measure of behavioral performance across all subjects. These findings support a crucial role of prediction error signals, likely originating from dopaminergic midbrain neurons, in enabling learning of action selection preferences on the basis of obtained rewards. Thus, spontaneously observed individual differences in decision making performance demonstrate the suggested dependence of this type of learning on the functional integrity of the dopaminergic striatal system in humans. 10.1523/JNEUROSCI.2496-07.2007

Neural Ensembles in CA3 Transiently Encode Paths Forward of the Animal at a Decision Point

Adam Johnson — 2007-11-08T13:38:39-00:00

J. Neurosci., Vol. 27, No. 45. (7 November 2007), pp. 12176-12189.

Neural ensembles were recorded from the CA3 region of rats running on T-based decision tasks. Examination of neural representations of space at fast time scales revealed a transient but repeatable phenomenon as rats made a decision: the location reconstructed from the neural ensemble swept forward, first down one path and then the other. Estimated representations were coherent and preferentially swept ahead of the animal rather than behind the animal, implying it represented future possibilities rather than recently traveled paths. Similar phenomena occurred at other important decisions (such as in recovery from an error). Local field potentials from these sites contained pronounced theta and gamma frequencies, but no sharp wave frequencies. Forward-shifted spatial representations were influenced by task demands and experience. These data suggest that the hippocampus does not represent space as a passive computation, but rather that hippocampal spatial processing is an active process likely regulated by cognitive mechanisms. 10.1523/JNEUROSCI.3761-07.2007

Anterior Prefrontal Function and the Limits of Human Decision-Making

Etienne Koechlin — 2007-10-26T08:53:52-00:00

Science, Vol. 318, No. 5850. (26 October 2007), pp. 594-598.

The frontopolar cortex (FPC), the most anterior part of the frontal lobes, forms the apex of the executive system underlying decision-making. Here, we review empirical evidence showing that the FPC function enables contingent interposition of two concurrent behavioral plans or mental tasks according to respective reward expectations, overcoming the serial constraint that bears upon the control of task execution in the prefrontal cortex. This function is mechanistically explained by interactions between FPC and neighboring prefrontal regions. However, its capacity appears highly limited, which suggests that the FPC is efficient for protecting the execution of long-term mental plans from immediate environmental demands and for generating new, possibly more rewarding, behavioral or cognitive sequences, rather than for complex decision-making and reasoning. 10.1126/science.1142995

Neural correlates of a postponed decision report

Luis Lemus — 2007-10-23T09:32:44-00:00

Proceedings of the National Academy of Sciences (16 October 2007), 0707961104.

Depending on environmental demands, a decision based on a sensory evaluation may be either immediately reported or postponed for later report. If postponed, the decision must be held in memory. But what exactly is stored by the underlying memory circuits, the final decision itself or the sensory information that led to it? Here, we report that, during a postponed decision report period, the activity of medial premotor cortex neurons encodes both the result of the sensory evaluation that corresponds to the monkey's possible choices and past sensory information on which the decision is based. These responses could switch back and forth with remarkable flexibility across the postponed decision report period. Moreover, these responses covaried with the animal's decision report. We propose that maintaining in working memory the original stimulus information on which the decision is based could serve to continuously update the postponed decision report in this task. 10.1073/pnas.0707961104

Prospect Theory: An Analysis of Decision under Risk

Daniel Kahneman — 2005-02-20T19:58:05-00:00

Econometrica, Vol. 47, No. 2. (1979), pp. 263-292.

This paper presents a critique of expected utility theory as a descriptive model of decision making under risk, and develops an alternative model, called prospect theory. Choices among risky prospects exhibit several pervasive effects that are inconsistent with the basic tenets of utility theory. In particular, people underweight outcomes that are merely probable in comparison with outcomes that are obtained with certainty. This tendency, called the certainty effect, contributes to risk aversion in choices involving sure gains and to risk seeking in choices involving sure losses. In addition, people generally discard components that are shared by all prospects under consideration. This tendency, called the isolation effect, leads to inconsistent preferences when the same choice is presented in different forms. An alternative theory of choice is developed, in which value is assigned to gains and losses rather than to final assets and in which probabilities are replaced by decision weights. The value function is normally concave for gains, commonly convex for losses, and is generally steeper for losses than for gains. Decision weights are generally lower than the corresponding probabilities, except in the range of low probabilities. Overweighting of low probabilities may contribute to the attractiveness of both insurance and gambling.

Learning the value of information in an uncertain world.

Timothy E J Behrens — 2007-08-14T13:37:12-00:00

Nat Neurosci (5 August 2007)

Our decisions are guided by outcomes that are associated with decisions made in the past. However, the amount of influence each past outcome has on our next decision remains unclear. To ensure optimal decision-making, the weight given to decision outcomes should reflect their salience in predicting future outcomes, and this salience should be modulated by the volatility of the reward environment. We show that human subjects assess volatility in an optimal manner and adjust decision-making accordingly. This optimal estimate of volatility is reflected in the fMRI signal in the anterior cingulate cortex (ACC) when each trial outcome is observed. When a new piece of information is witnessed, activity levels reflect its salience for predicting future outcomes. Furthermore, variations in this ACC signal across the population predict variations in subject learning rates. Our results provide a formal account of how we weigh our different experiences in guiding our future actions.

Temporal Filtering of Reward Signals in the Dorsal Anterior Cingulate Cortex during a Mixed-Strategy Game

Hyojung Seo — 2007-08-03T09:46:31-00:00

J. Neurosci., Vol. 27, No. 31. (1 August 2007), pp. 8366-8377.

The process of decision making in humans and other animals is adaptive and can be tuned through experience so as to optimize the outcomes of their choices in a dynamic environment. Previous studies have demonstrated that the anterior cingulate cortex plays an important role in updating the animal's behavioral strategies when the action outcome contingencies change. Moreover, neurons in the anterior cingulate cortex often encode the signals related to expected or actual reward. We investigated whether reward-related activity in the anterior cingulate cortex is affected by the animal's previous reward history. This was tested in rhesus monkeys trained to make binary choices in a computer-simulated competitive zero-sum game. The animal's choice behavior was relatively close to the optimal strategy but also revealed small systematic biases that are consistent with the use of a reinforcement learning algorithm. In addition, the activity of neurons in the dorsal anterior cingulate cortex that was related to the reward received by the animal in a given trial often was modulated by the rewards in the previous trials. Some of these neurons encoded the rate of rewards in previous trials, whereas others displayed activity modulations more closely related to the reward prediction errors. In contrast, signals related to the animal's choices were represented only weakly in this cortical area. These results suggest that neurons in the dorsal anterior cingulate cortex might be involved in the subjective evaluation of choice outcomes based on the animal's reward history. 10.1523/JNEUROSCI.2369-07.2007

Understanding Neural Coding through the Model-Based Analysis of Decision Making

Greg Corrado — 2007-08-03T09:38:54-00:00

J. Neurosci., Vol. 27, No. 31. (1 August 2007), pp. 8178-8180.

The study of decision making poses new methodological challenges for systems neuroscience. Whereas our traditional approach linked neural activity to external variables that the experimenter directly observed and manipulated, many of the key elements that contribute to decisions are internal to the decider. Variables such as subjective value or subjective probability may be influenced by experimental conditions and manipulations but can neither be directly measured nor precisely controlled. Pioneering work on the neural basis of decision circumvented this difficulty by studying behavior in static conditions, in which knowledge of the average state of these quantities was sufficient. More recently, a new wave of studies has confronted the conundrum of internal decision variables more directly by leveraging quantitative behavioral models. When these behavioral models are successful in predicting a subject's choice, the model's internal variables may serve as proxies for the unobservable decision variables that actually drive behavior. This new methodology has allowed researchers to localize neural subsystems that encode hidden decision variables related to free choice and to study these variables under dynamic conditions. 10.1523/JNEUROSCI.1590-07.2007

Functional Specialization of the Primate Frontal Cortex during Decision Making

Daeyeol Lee — 2007-08-03T09:16:35-00:00

J. Neurosci., Vol. 27, No. 31. (1 August 2007), pp. 8170-8173.

Economic theories of decision making are based on the principle of utility maximization, and reinforcement-learning theory provides computational algorithms that can be used to estimate the overall reward expected from alternative choices. These formal models not only account for a large range of behavioral observations in human and animal decision makers, but also provide useful tools for investigating the neural basis of decision making. Nevertheless, in reality, decision makers must combine different types of information about the costs and benefits associated with each available option, such as the quality and quantity of expected reward and required work. In this article, we put forward the hypothesis that different subdivisions of the primate frontal cortex may be specialized to focus on different aspects of dynamic decision-making processes. In this hypothesis, the lateral prefrontal cortex is primarily involved in maintaining the state representation necessary to identify optimal actions in a given environment. In contrast, the orbitofrontal cortex and the anterior cingulate cortex might be primarily involved in encoding and updating the utilities associated with different sensory stimuli and alternative actions, respectively. These cortical areas are also likely to contribute to decision making in a social context. 10.1523/JNEUROSCI.1561-07.2007

The Role of the Dorsal Striatum in Reward and Decision-Making

Bernard Balleine — 2007-08-03T09:08:38-00:00

J. Neurosci., Vol. 27, No. 31. (1 August 2007), pp. 8161-8165.

Although the involvement in the striatum in the refinement and control of motor movement has long been recognized, recent description of discrete frontal corticobasal ganglia networks in a range of species has focused attention on the role particularly of the dorsal striatum in executive functions. Current evidence suggests that the dorsal striatum contributes directly to decision-making, especially to action selection and initiation, through the integration of sensorimotor, cognitive, and motivational/emotional information within specific corticostriatal circuits involving discrete regions of striatum. We review key evidence from recent studies in rodent, nonhuman primate, and human subjects. 10.1523/JNEUROSCI.1554-07.2007

Reinforcement learning signals predict future decisions.

MX Cohen — 2007-02-21T01:41:58-00:00

J Neurosci, Vol. 27, No. 2. (10 January 2007), pp. 371-378.

Optimal behavior in a competitive world requires the flexibility to adapt decision strategies based on recent outcomes. In the present study, we tested the hypothesis that this flexibility emerges through a reinforcement learning process, in which reward prediction errors are used dynamically to adjust representations of decision options. We recorded event-related brain potentials (ERPs) while subjects played a strategic economic game against a computer opponent to evaluate how neural responses to outcomes related to subsequent decision-making. Analyses of ERP data focused on the feedback-related negativity (FRN), an outcome-locked potential thought to reflect a neural prediction error signal. Consistent with predictions of a computational reinforcement learning model, we found that the magnitude of ERPs after losing to the computer opponent predicted whether subjects would change decision behavior on the subsequent trial. Furthermore, FRNs to decision outcomes were disproportionately larger over the motor cortex contralateral to the response hand that was used to make the decision. These findings provide novel evidence that humans engage a reinforcement learning process to adjust representations of competing decision options.

Reinforcement learning and decision making in monkeys during a competitive game

Daeyeol Lee — 2005-02-01T18:09:11-00:00

Cognitive Brain Research, Vol. 22, No. 1. (December 2004), pp. 45-58.

Animals living in a dynamic environment must adjust their decision-making strategies through experience. To gain insights into the neural basis of such adaptive decision-making processes, we trained monkeys to play a competitive game against a computer in an oculomotor free-choice task. The animal selected one of two visual targets in each trial and was rewarded only when it selected the same target as the computer opponent. To determine how the animal's decision-making strategy can be affected by the opponent's strategy, the computer opponent was programmed with three different algorithms that exploited different aspects of the animal's choice and reward history. When the computer selected its targets randomly with equal probabilities, animals selected one of the targets more often, violating the prediction of probability matching, and their choices were systematically influenced by the choice history of the two players. When the computer exploited only the animal's choice history but not its reward history, animal's choice became more independent of its own choice history but was still related to the choice history of the opponent. This bias was substantially reduced, but not completely eliminated, when the computer used the choice history of both players in making its predictions. These biases were consistent with the predictions of reinforcement learning, suggesting that the animals sought optimal decision-making strategies using reinforcement learning algorithms.

Cocaine-induced decision-making deficits are mediated by miscoding in basolateral amygdala.

Thomas A Stalnaker — 2007-07-11T07:45:25-00:00

Nat Neurosci (1 July 2007)

Addicts and drug-experienced animals have decision-making deficits in reversal-learning tasks and more complex 'gambling' variants. Here we show evidence that these deficits are mediated by persistent encoding of outdated associative information in the basolateral amygdala. Cue-selective neurons in the basolateral amygdala, recorded in cocaine-treated rats, failed to change cue preference during reversal learning. Further, the presence of these neurons was critical to the expression of the reversal-learning deficit in the cocaine-treated rats.

Adaptive decision making and value in the anterior cingulate cortex

Mark Walton — 2007-07-23T12:19:50-00:00

NeuroImage, Vol. 36, No. Supplement 2. (2007), pp. T142-T154.

Choosing an appropriate response in an uncertain and varying world is central to adaptive behaviour. The frequent activation of the anterior cingulate cortex (ACC) in a diverse range of tasks has lead to intense interest in and debate over its role in the guidance and control of performance. Here, we consider how this issue can be informed by a series of studies considering the ACC's role in more naturalistic situations where there is no single certain correct response and the relationships between choices and their consequences vary. A neuroimaging study of response switching demonstrates that dorsal ACC is not simply concerned with self-generated responses or error monitoring in isolation, but is instead involved in evaluating the outcome of choices, positive or negative, that have been voluntarily chosen. By contrast, an interconnected part of the orbitofrontal cortex is shown to be more active when attending to consequences of actions instructed by the experimenter. This dissociation is explained with reference to the anatomy of these regions in humans as demonstrated by diffusion weighted imaging. Lesions to a corresponding ACC region in monkeys has no effect on animals' ability to detect or immediately correct errors when response contingencies reverse, but renders them unable to sustain appropriate behaviour due to an impairment in the ability to integrate over time their recent history of choices and outcomes. Taken together, this implies a prominent role for the ACC within a distributed network of regions that determine the dynamic value of actions and guide decision making appropriately.

Bees in two-armed bandit situations: foraging choices and possible decision mechanisms

Tamar Keasar — 2007-01-04T02:58:43-00:00

Behav. Ecol., Vol. 13, No. 6. (1 November 2002), pp. 757-765.

In multi-armed bandit situations, gamblers must choose repeatedly between options that differ in reward probability, without prior information on the options' relative profitability. Foraging bumblebees encounter similar situations when choosing repeatedly among flower species that differ in food rewards. Unlike proficient gamblers, bumblebees do not choose the highest-rewarding option exclusively. This incomplete exclusiveness may reflect an adaptive sampling strategy. A cost--benefit analysis predicts decreased sampling levels with increasing differences in mean profitability between the available food sources. We simulated two-armed bandit situations in laboratory experiments to test this prediction. Bumblebees (Bombus terrestris L.) made 300 foraging visits to blue and yellow artificial flowers that dispensed sucrose solution according to seven probabilistic reward schedules. Reward schedules varied in profitability differences between the two feeding options. As predicted, the bees specialized more on the higher-rewarding food type (and thus sampled the alternative less) when the mean reward difference between the feeding options was larger. Choice ratios of individual bees were linearly related to the reward ratios they had experienced. It has been suggested that the behavioral mechanism underlying incomplete exclusiveness may involve simple rules of thumb that do not require long-term memory. However, the bees' response to recent foraging experience (rewarded and non-rewarded visits) differed between the beginning and the end of observation sessions and between treatments. Simulations of the Rescorla-Wagner difference learning rule reproduced the main trends of the results. These findings suggest that the observed incomplete exclusiveness results from associative learning involving long-term memory. 10.1093/beheco/13.6.757

Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control

Nathaniel Daw — 2005-11-23T18:52:20-00:00

Nature Neuroscience, Vol. 8, No. 12. (06 November 2005), pp. 1704-1711.

Via Freedom to Coercion: The Emergence of Costly Punishment

Christoph Hauert — 2007-06-29T08:52:25-00:00

Science, Vol. 316, No. 5833. (29 June 2007), pp. 1905-1907.

In human societies, cooperative behavior in joint enterprises is often enforced through institutions that impose sanctions on defectors. Many experiments on so-called public goods games have shown that in the absence of such institutions, individuals are willing to punish defectors, even at a cost to themselves. Theoretical models confirm that social norms prescribing the punishment of uncooperative behavior are stable--once established, they prevent dissident minorities from spreading. But how can such costly punishing behavior gain a foothold in the population? A surprisingly simple model shows that if individuals have the option to stand aside and abstain from the joint endeavor, this paves the way for the emergence and establishment of cooperative behavior based on the punishment of defectors. Paradoxically, the freedom to withdraw from the common enterprise leads to enforcement of social norms. Joint enterprises that are compulsory rather than voluntary are less likely to lead to cooperation. 10.1126/science.1141588

Dopamine-Mushroom Body Circuit Regulates Saliency-Based Decision-Making in Drosophila

Ke Zhang — 2007-06-29T08:23:23-00:00

Science, Vol. 316, No. 5833. (29 June 2007), pp. 1901-1904.

Drosophila melanogaster can make appropriate choices among alternative flight options on the basis of the relative salience of competing visual cues. We show that this choice behavior consists of early and late phases; the former requires activation of the dopaminergic system and mushroom bodies, whereas the latter is independent of these activities. Immunohistological analysis showed that mushroom bodies are densely innervated by dopaminergic axons. Thus, the circuit from the dopamine system to mushroom bodies is crucial for choice behavior in Drosophila. 10.1126/science.1137357

Functional organization of the medial frontal cortex.

MF Rushworth — 2007-05-16T16:00:03-00:00

Curr Opin Neurobiol, Vol. 17, No. 2. (April 2007), pp. 220-227.

The anterior cingulate cortex (ACC) and adjacent areas of the medial frontal cortex (MFC) have been implicated in monitoring behaviour and in detecting errors. Recent evidence, however, suggests that the ACC not only registers the occurrence of errors but also represents other aspects of the reinforcement history that are crucial for guiding behaviour. Other studies raise the possibility that dorsal MFC areas not only monitor behaviour but also actually control response selection, particularly when the task in hand is changing. Many decisions are made in social contexts and their chances of success depend on what other individuals are doing. Evaluation of other individuals is therefore crucial for effective action selection, and some ACC regions are implicated in this process.

Orbitofrontal Cortex and Its Contribution to Decision-Making.

Jonathan D Wallis — 2007-05-04T15:29:59-00:00

Annu Rev Neurosci (6 April 2007)

Damage to orbitofrontal cortex (OFC) produces an unusual pattern of deficits. Patients have intact cognitive abilities but are impaired in making everyday decisions. Here we review anatomical, neuropsychological, and neurophysiological evidence to determine the neuronal mechanisms that might underlie these impairments. We suggest that OFC plays a key role in processing reward: It integrates multiple sources of information regarding the reward outcome to derive a value signal. In effect, OFC calculates how rewarding a reward is. This value signal can then be held in working memory where it can be used by lateral prefrontal cortex to plan and organize behavior toward obtaining the outcome, and by medial prefrontal cortex to evaluate the overall action in terms of its success and the effort that was required. Thus, acting together, these prefrontal areas can ensure that our behavior is most efficiently directed towards satisfying our needs. Expected online publication date for the Annual Review of Neuroscience Volume 30 is June 16, 2007. Please see http://www.annualreviews.org/catalog/pubdates.aspx for revised estimates.

Neural correlates of decision variables in parietal cortex.

ML Platt — 2005-02-01T19:51:55-00:00

Nature, Vol. 400, No. 6741. (15 July 1999), pp. 233-238.

Decision theory proposes that humans and animals decide what to do in a given situation by assessing the relative value of each possible response. This assessment can be computed, in part, from the probability that each action will result in a gain and the magnitude of the gain expected. Here we show that the gain (or reward) a monkey can expect to realize from an eye-movement response modulates the activity of neurons in the lateral intraparietal area, an area of primate cortex that is thought to transform visual signals into eye-movement commands. We also show that the activity of these neurons is sensitive to the probability that a particular response will result in a gain. When animals can choose freely between two alternative responses, the choices subjects make and neuronal activation in this area are both correlated with the relative amount of gain that the animal can expect from each response. Our data indicate that a decision-theoretic model may provide a powerful new framework for studying the neural processes that intervene between sensation and action.

The Neural Basis of Loss Aversion in Decision-Making Under Risk

Sabrina Tom — 2007-01-30T10:47:34-00:00

Science, Vol. 315, No. 5811. (26 January 2007), pp. 515-518.

People typically exhibit greater sensitivity to losses than to equivalent gains when making decisions. We investigated neural correlates of loss aversion while individuals decided whether to accept or reject gambles that offered a 50/50 chance of gaining or losing money. A broad set of areas (including midbrain dopaminergic regions and their targets) showed increasing activity as potential gains increased. Potential losses were represented by decreasing activity in several of these same gain-sensitive areas. Finally, individual differences in behavioral loss aversion were predicted by a measure of neural loss aversion in several regions, including the ventral striatum and prefrontal cortex. 10.1126/science.1134239

Cortical substrates for exploratory decisions in humans

Nathaniel Daw — 2006-06-20T11:57:29-00:00

Nature, Vol. 441, No. 7095., pp. 876-879.

Choosing the Lesser of Two Evils, the Better of Two Goods: Specifying the Roles of Ventromedial Prefrontal Cortex and Dorsal Anterior Cingulate in Object Choice

Karina Blair — 2006-11-03T10:19:20-00:00

J. Neurosci., Vol. 26, No. 44. (1 November 2006), pp. 11379-11386.

The ventromedial prefrontal cortex (vmPFC) and dorsal anterior cingulate cortices (ACd) are considered important for reward-based decision making. However, work distinguishing their individual functional contributions has only begun. One aspect of decision making that has received little attention is that making the right choice often translates to making the better choice. Thus, response choice often occurs in situations where both options are desirable (e.g., choosing between mousse au chocolat or creme caramel cheesecake from a menu) or, alternatively, in situations where both options are undesirable. Moreover, response choice is easier when the reinforcements associated with the objects are far apart, rather than close together, in value. We used functional magnetic resonance imaging to delineate the functional roles of the vmPFC and ACd by investigating these two aspects of decision making: (1) decision form (i.e., choosing between two objects to gain the greater reward or the lesser punishment), and (2) between-object reinforcement distance (i.e., the difference in reinforcements associated with the two objects). Blood oxygen level-dependent (BOLD) responses within the ACd and vmPFC were both related to decision form but differentially. Whereas ACd showed greater responses when deciding between objects to gain the lesser punishment, vmPFC showed greater responses when deciding between objects to gain the greater reward. Moreover, vmPFC was sensitive to reinforcement expectations associated with both the chosen and the forgone choice. In contrast, BOLD responses within ACd, but not vmPFC, related to between-object reinforcement distance, increasing as the distance between the reinforcements of the two objects decreased. These data are interpreted with reference to models of ACd and vmPFC functioning. 10.1523/JNEUROSCI.1640-06.2006

Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning)

Richard Sutton — 2005-03-02T20:12:18-00:00

(01 March 1998)

Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In <i>Reinforcement Learning</i>, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability.<br /> <br /> The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

Computational algorithms and neuronal network models underlying decision processes

Yutaka Sakai — 2006-08-31T10:00:55-00:00

Neural Networks, Vol. In Press, Corrected Proof

Animals or humans often encounter such situations in which they must choose their behavioral responses to be made in the near or distant future. Such a decision is made through continuous and bidirectional interactions between the environment surrounding the brain and its internal state or dynamical processes. Therefore, decision making may provide a unique field of researches for studying information processing by the brain, a biological system open to information exchanges with the external world. To make a decision, the brain must analyze pieces of information given externally, past experiences in a similar situation, possible behavioral responses, and predicted outcomes of the individual responses. In this article, we review results of recent experimental and theoretical studies of neuronal substrates and computational algorithms for decision processes.

Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans: Nature

2006-08-24T10:14:48-00:00

Midbrain dopamine neurons encode decisions for future action

Genela Morris — 2006-07-26T12:30:16-00:00

Nature Neuroscience, Vol. 9, No. 8. (23 July 2006), pp. 1057-1063.

A re-examination of probability matching and rational choice

David Shanks — 2005-10-16T16:52:31-00:00

Journal of Behavioral Decision Making, Vol. 15, No. 3. (18 March 2002), pp. 233-250.

In a typical probability learning task participants are presented with a repeated choice between two response alternatives, one of which has a higher payoff probability than the other. Rational choice theory requires that participants should eventually allocate all their responses to the high-payoff alternative, but previous research has found that people fail to maximize their payoffs. Instead, it is commonly observed that people match their response probabilities to the payoff probabilities. We report three experiments on this choice anomaly using a simple probability learning task in which participants were provided with (i) large financial incentives, (ii) meaningful and regular feedback, and (iii) extensive training. In each experiment large proportions of participants adopted the optimal response strategy and all three of the factors mentioned above contributed to this. The results are supportive of rational choice theory. Copyright © 2002 John Wiley & Sons, Ltd.