makimaki's Top Recommendations

Abstract

Decision theory proposes that humans and animals decide what to do in a given situation by assessing the relative value of each possible response. This assessment can be computed, in part, from the probability that each action will result in a gain and the magnitude of the gain expected. Here we show that the gain (or reward) a monkey can expect to realize from an eye-movement response modulates the activity of neurons in the lateral intraparietal area, an area of primate ...

Abstract

Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In _Reinforcement Learning_, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts ...

Abstract

Uncertainty is critical in the measure of information and in assessing the accuracy of predictions. It is determined by probability P, being maximal at P = 0.5 and decreasing at higher and lower probabilities. Using distinct stimuli to indicate the probability of reward, we found that the phasic activation of dopamine neurons varied monotonically across the full range of probabilities, supporting past claims that this response codes the discrepancy between predicted and actual reward. In contrast, a previously unobserved response covaried ...

Abstract

It is important for animals to estimate the value of rewards as accurately as possible. Because the number of potential reward values is very large, it is necessary that the brain's limited resources be allocated so as to discriminate better among more likely reward outcomes at the expense of less likely outcomes. We found that midbrain dopamine neurons rapidly adapted to the information provided by reward-predicting stimuli. Responses shifted relative to the expected reward value, and the gain adjusted to the ...

Abstract

Psychologists and economists have long appreciated the contribution of reward history and expectation to decision-making. Yet we know little about how specific histories of choice and reward lead to an internal representation of the "value" of possible actions. We approached this problem through an integrated application of behavioral, computational, and physiological techniques. Monkeys were placed in a dynamic foraging environment in which they had to track the changing values of alternative choices through time. In this context, the monkeys' foraging behavior ...

Abstract

Since its conception three decades ago, the idea that the striatum consists of a dorsal sensorimotor part and a ventral portion processing limbic information has sparked a quest for functional correlates and anatomical characteristics of the striatal divisions. But this classic dorsal-ventral distinction might not offer the best view of striatal function. Anatomy and neurophysiology show that the two striatal areas have the same basic structure and that sharp boundaries are absent. Behaviorally, a distinction between dorsolateral and ventromedial seems most ...

Abstract

In this article we develop the concept that the hippocampus and the midbrain dopaminergic neurons of the ventral tegmental area (VTA) form a functional loop. Activation of the loop begins when the hippocampus detects newly arrived information that is not already stored in its long-term memory. The resulting novelty signal is conveyed through the subiculum, accumbens, and ventral pallidum to the VTA where it contributes (along with salience and goal information) to the novelty-dependent firing of these cells. In the upward ...

Abstract

Behavioral studies suggest that making a decision involves representing the overall desirability of all available actions and then selecting that action that is most desirable. Physiological studies have proposed that neurons in the parietal cortex play a role in selecting movements for execution. To test the hypothesis that these parietal neurons encode the subjective desirability of making particular movements, we exploited Nash's game theoretic equilibrium, during which the subjective desirability of multiple actions should be equal for human players. Behavior measured ...

Abstract

There is substantial evidence that dopamine is involved in reward learning and appetitive conditioning. However, the major reinforcement learning-based theoretical models of classical conditioning (crudely, prediction learning) are actually based on rules designed to explain instrumental conditioning (action learning). Extensive anatomical, pharmacological, and psychological data, particularly concerning the impact of motivational manipulations, show that these models are unreasonable. We review the data and consider the involvement of a rich collection of different neural systems in various aspects of these forms of ...

Abstract

Midbrain dopamine and striatal tonically active neurons (TANs, presumed acetylcholine interneurons) signal behavioral significance of environmental events. Since striatal dopamine and acetylcholine affect plasticity of cortico-striatal transmission and are both crucial to learning, they may serve as teachers in the basal ganglia circuits. We recorded from both neuronal populations in monkeys performing a probabilistic instrumental conditioning task. Both neuronal types respond robustly to reward-related events. Although different events yielded responses with different latencies, the responses of the two populations coincided, indicating ...

Abstract

Decision-making behavior has been studied extensively, but the neurophysiological mechanisms responsible for this remarkable cognitive ability are just beginning to be understood. Here we propose neural computations that can account for the formation of categorical decisions about sensory stimuli by accumulating information over time into a single quantity: the logarithm of the likelihood ratio favoring one alternative over another. We also review electrophysio-logical studies that have identified brain structures that may be involved in computing this sort of decision variable. The ...

Abstract

Expectation of reward motivates our behaviors and influences our decisions. Indeed, neuronal activity in many brain areas is modulated by expected reward. However, it is still unclear where and how the reward-dependent modulation of neuronal activity occurs and how the reward-modulated signal is transformed into motor outputs. Recent studies suggest an important role of the basal ganglia. Sensorimotor/cognitive activities of neurons in the basal ganglia are strongly modulated by expected reward. Through their abundant outputs to the brain stem motor areas ...

Abstract

Movement selection depends on the outcome of prior behavior. Posterior cingulate cortex (CGp) is strongly connected with both limbic and oculomotor circuitry, and CGp neurons respond following saccades, suggesting a role in signaling the motivational outcome of gaze shifts. To test this hypothesis, single CGp neurons were studied in monkeys while they shifted gaze to visual targets for liquid rewards that varied in size or were delivered probabilistically. CGp neurons responded following saccades as well as following reward delivery, and these ...

Abstract

The field of basal ganglia research is exploding on every level - from discoveries at the molecular level to those based on human brain imaging. A remarkable series of new findings support the view that the basal ganglia are essential for some forms of learning-related plasticity. Other new findings are challenging some of the basic tenets of the field as it now stands. Combined with the new evidence on learning-related functions of the basal ganglia, these studies suggest that the basal ...

Abstract

Midbrain dopamine neurons are key components of the brain's reward system, which is thought to guide reward-seeking behaviours. Although recent studies have shown how dopamine neurons respond to rewards and sensory stimuli predicting reward, it is unclear which parts of the brain provide dopamine neurons with signals necessary for these actions. Here we show that the primate lateral habenula, part of the structure called the epithalamus, is a major candidate for a source of negative reward-related signals in dopamine neurons. We ...

Abstract

Visual stimuli can acquire positive or negative value through their association with rewards and punishments, a process called reinforcement learning. Although we now know a great deal about how the brain analyses visual information, we know little about how visual representations become linked with values. To study this process, we turned to the amygdala, a brain structure implicated in reinforcement learning. We recorded the activity of individual amygdala neurons in monkeys while abstract images acquired either positive or negative value through ...

Abstract

The midbrain dopamine neurons are hypothesized to provide a physiological correlate of the reward prediction error signal required by current models of reinforcement learning. We examined the activity of single dopamine neurons during a task in which subjects learned by trial and error when to make an eye movement for a juice reward. We found that these neurons encoded the difference between the current reward and a weighted average of previous rewards, a reward prediction error, but only for outcomes that ...

Abstract

The capacity to anticipate the timing of environmental cues allows us to allocate sensory resources at the right time and prepare actions. Such anticipation requires knowledge of elapsed time and of the probability that an event will occur. Here we show that neurons in the parietal cortex represent the probability, as a function of time, that a salient event is likely to occur. Rhesus monkeys were trained to make eye movements to peripheral targets after a light dimmed. Within a block ...

Abstract

The anterior cingulate and orbital cortices and the ventral striatum process different aspects of reward evaluation, whereas the dorsolateral prefrontal cortex and the dorsal striatum are involved in cognitive function. Collectively, these areas are critical to decision making. We mapped the striatal area that receives information about reward evaluation. We also explored the extent to which terminals from reward-related cortical areas converge in the striatum with those from cognitive regions. Using three-dimensional-rendered reconstructions of corticostriatal projection fields along with two-dimensional chartings, ...

Abstract

. This article introduces a class of incremental learning procedures specialized for prediction---that is, for using past experience with an incompletely known system to predict its future behavior. Whereas conventional prediction-learning methods assign credit by means of the difference between predicted and actual outcomes, the new methods assign credit by means of the difference between temporally successive predictions. Although such temporal-difference methods have been used in Samuel's... ...

Abstract

Current models of the basal ganglia and dopamine neurons emphasize their role in reinforcement learning. However, the role of dopamine neurons in decision making is still unclear. We recorded from dopamine neurons in monkeys engaged in two types of trial: reference trials in an instructed-choice task and decision trials in a two-armed bandit decision task. We show that the activity of dopamine neurons in the decision setting is modulated according to the value of the upcoming action. Moreover, analysis of the ...

Abstract

Much is known about how people make decisions under varying levels of probability (risk). Less is known about the neural basis of decision-making when probabilities are uncertain because of missing information (ambiguity). In decision theory, ambiguity about probabilities should not affect choices. Using functional brain imaging, we show that the level of ambiguity in choices correlates positively with activation in the amygdala and orbitofrontal cortex, and negatively with a striatal system. Moreover, striatal activity correlates positively with expected reward. Neurological subjects ...

Abstract

Addictive drugs have been hypothesized to access the same neurophysiological mechanisms as natural learning systems. These natural learning systems can be modeled through temporal-difference reinforcement learning (TDRL), which requires a reward-error signal that has been hypothesized to be carried by dopamine. TDRL learns to predict reward by driving that reward-error signal to zero. By adding a noncompensable drug-induced dopamine increase to a TDRL model, a computational model of addiction is constructed that over-selects actions leading to drug receipt. The model provides ...

Abstract

1. Posterior cingulate cortex, although widely regarded as a part of the limbic system, is connected most strongly to parietal and frontal areas with sensory, motor, and cognitive functions. To gain insight into the functional nature of posterior cingulate cortex, we have recorded from its neurons in monkeys performing oculomotor tasks known to activate parietal and frontal neurons. We have found that posterior cingulate neurons fire during periods of ocular fixation at a rate determined by the angle of gaze and ...

Abstract

The function of the phasic dopamine signal, seen in response to salient and rewarding stimuli, has been heavily debated. The reward prediction error hypothesis has been criticised for the suggestion that such a complex signal could be derived at short latencies, relying only on subcortical inputs. However, as more has been learnt about the nature of the subcortical inputs, we are led to challenge this criticism. Here we suggest that the subcortical inputs can indeed support complex calculations and that it ...

Abstract

Neural activity signifying the expectation of reward has been found recently in many parts of the brain, including midbrain and cortical structures. These signals can facilitate goal-directed behavior or the learning of new skills based on reinforcements. Here we show that neurons in the supplementary motor area (SMA), an area concerned with movements of the body and limbs, also carry a reward expectancy signal in the post-saccadic period of oculomotor tasks. While the monkeys performed blocks of memory-guided and object-based saccades, ...

Abstract

Many decisions are made under uncertainty; that is, with limited information about their potential consequences. Previous neuroimaging studies of decision making have implicated regions of the medial frontal lobe in processes related to the resolution of uncertainty. However, a different set of regions in dorsal prefrontal and posterior parietal cortices has been reported to be critical for selection of actions to unexpected or unpredicted stimuli within a sequence. In the current study, we induced uncertainty using a novel task that required ...

Abstract

Unexpected, biologically salient stimuli elicit a short-latency, phasic response in midbrain dopaminergic (DA) neurons. Although this signal is important for reinforcement learning, the information it conveys to forebrain target structures remains uncertain. One way to decode the phasic DA signal would be to determine the perceptual properties of sensory inputs to DA neurons. After local disinhibition of the superior colliculus in anesthetized rats, DA neurons became visually responsive, whereas disinhibition of the visual cortex was ineffective. As the primary source of ...

Abstract

SummaryImpulsivity and loss of self-control in drug-dependent patients have been associated with the manner in which they discount delayed rewards. Although drugs of abuse have been shown to modify perceived time-duration, little is known regarding the relationship between impulsive decision-making in intertemporal choice and estimation of time-duration.In classical economic theory, it has been hypothesized that people discount future reward value exponentially. In exponential discounting, a temporal discounting rate is constant over time, which has been referred to as dynamic consistency. However, ...

Abstract

We study the behavior of subjects facing choices between certain, risky, and ambiguous lotteries. Subjects' choices are consistent with the economic theories modeling ambiguity aversion. Our results support the conjecture that subjects face choice tasks as an estimation of the value of the lotteries, and that the difficulty of the choice is an important explanatory variable (in addition to risk and ambiguity aversion).The brain imaging data suggest that such estimation is of an approximate nature when the choices involve ambiguous and ...

Abstract

Intelligent behaviour requires self-control based on the consequences of actions. The countermanding task is designed to study self-control; it requires subjects to withhold planned movements in response to an imperative stop signal, which they can do with varying success. In humans, the medial frontal cortex has been implicated in the supervisory control of action. In monkeys, the supplementary eye field in the dorsomedial frontal cortex is involved in producing eye movements, but its precise function has not been clarified. To investigate ...

Abstract

A large class of statistical decision models for performance in simple information processing tasks can be described by linear, first-order, stochastic differential equations (SDEs), whose solutions are diffusion processes. In such models, the first passage time for the diffusion process through a response criterion determines the time at which an observer makes a decision about the identity of a stimulus. Because the assumptions of many cognitive models lead to SDEs that are time inhomogeneous, classical methods for solving such first passage ...

Abstract

Mesostriatal dopaminergic neurons (DANs) and striatal cholinergic neurons (tonically active neurons, TANs) participate in signalling the behavioural or reward-related significance of stimuli in the environment. An antagonistic balance between dopamine (DA) and ACh is well known to regulate postsynaptic signal integration in the striatum. Recent findings have revealed additional presynaptic ACh-DA interactions of previously unappreciated sophistication. Striatal ACh acts presynaptically to polarize powerfully how opposing DAN activities are transduced into DA release. Furthermore, characteristic reward-related activities of TANs and DANs are ...

Abstract

Neurons in a small number of brain structures detect rewards and reward-predicting stimuli and are active during the expectation of predictable food and liquid rewards. These neurons code the reward information according to basic terms of various behavioural theories that seek to explain reward-directed learning, approach behaviour and decision-making. The involved brain structures include groups of dopamine neurons, the striatum including the nucleus accumbens, the orbitofrontal cortex and the amygdala. The reward information is fed to brain structures involved in decision-making ...

Abstract

BACKGROUND: Delays between actions and their outcomes severely hinder reinforcement learning systems, but little is known of the neural mechanism by which animals overcome this problem and bridge such delays. The nucleus accumbens core (AcbC), part of the ventral striatum, is required for normal preference for a large, delayed reward over a small, immediate reward (self-controlled choice) in rats, but the reason for this is unclear. We investigated the role of the AcbC in learning a free-operant instrumental response using delayed ...

Abstract

Contrary to conventional wisdom, it is not always advantageous to engage in thorough conscious deliberation before choosing. On the basis of recent insights into the characteristics of conscious and unconscious thought, we tested the hypothesis that simple choices (such as between different towels or different sets of oven mitts) indeed produce better results after conscious thought, but that choices in complex matters (such as between different houses or different cars) should be left to unconscious thought. Named the "deliberation-without-attention" hypothesis, it ...

Abstract

For many years, the basal ganglia were described in anatomy courses as strictly motor structures. Certainly, some of the most obvious and debilitating symptoms shown by persons with basal ganglia disorders are problems in motor control. However, the basal ganglia are not limited to motoric aspects of behavior: recent research shows that they are involved in most areas of cognitive and emotional functioning, consistent with their anatomical connections with all areas of the cortex. This review will focus on the roles ...

Abstract

SummaryUncertainty in various forms plagues our interactions with the environment. In a Bayesian statistical framework, optimal inference and prediction, based on unreliable observations in changing contexts, require the representation and manipulation of different forms of uncertainty. We propose that the neuromodulators acetylcholine and norepinephrine play a major role in the brain's implementation of these uncertainty computations. Acetylcholine signals expected uncertainty, coming from known unreliability of predictive cues within a context. Norepinephrine signals unexpected uncertainty, as when unsignaled context switches produce strongly ...

Abstract

The capacity to predict future events permits a creature to detect, model, and manipulate the causal structure of its interactions with its environment. Behavioral experiments suggest that learning is driven by changes in the expectations about future salient events such as rewards and punishments. Physiological work has recently complemented these studies by identifying dopaminergic neurons in the primate whose fluctuating output apparently signals changes or errors in the predictions of future salient and rewarding events. Taken together, these findings can be ...

Abstract

Our brains allow us to reason about alternatives and to make choices that are likely to pay off. Often there is no one correct answer, but instead one that is favoured simply because it is more likely to lead to reward. A variety of probabilistic classification tasks probe the covert strategies that humans use to decide among alternatives based on evidence that bears only probabilistically on outcome. Here we show that rhesus monkeys can also achieve such reasoning. We have trained ...

Abstract

Individuals value information that improves decision making. When social interactions complicate the decision process, acquiring information about others should be particularly valuable []. In primate societies, kinship, dominance, and reproductive status regulate social interactions [] and should therefore systematically influence the value of social information, but this has never been demonstrated. Here, we show that monkeys differentially value the opportunity to acquire visual information about particular classes of social images. Male rhesus macaques sacrificed fluid for the opportunity to view female ...

Abstract

Previous neurophysiological studies have reported that neurons in posterior cingulate cortex (PCC) respond after eye movements, and that these responses may vary with ambient illumination. In monkeys, PCC neurons also respond after the illumination of large visual patterns but not after the illumination of small visual targets on either reflexive saccade tasks or peripheral attention tasks. These observations suggest that neuronal activity in PCC is modulated by behavioral context, which varies with the timing and spatial distribution of visual and oculomotor ...

Abstract

Over the past two decades significant progress has been made toward understanding the neural basis of primate decision making, the biological process that combines sensory data with stored information to select and execute behavioral responses. The most striking progress in this area has been made in studies of visual-saccadic decision making, a system that is becoming a model for understanding decision making in general. In this system, theoretical models of efficient decision making developed in the social sciences are beginning to ...

Abstract

Imagine the decisions you might make while playing a simple game like 'matching pennies'. At each play, you and your opponent, say the mathematician John vonNeumann, each lay down a penny heads or tails up. If both pennies show the same side, vonNeumann wins, if not, you win. Before each play, you have the subjective experience of deciding what to do: of choosing whether to play heads or tails. Although decisions like these are not yet understood at a physiological level, ...

Abstract

Dopamine neurons play a key role in reward-related behaviors. Reward coding theories predict that dopamine neurons will be inhibited by or will not respond to aversive stimuli. Paradoxically, between 3 and 49% of presumed dopamine neurons are excited by aversive stimuli. We found that, in the ventral tegmental area of anesthetized rats, the population of presumed dopamine neurons that are excited by aversive stimuli is actually not dopaminergic. The identified dopamine neurons were inhibited by the aversive stimulus. These findings suggest ...

Abstract

The histories of the terms "reward" and "reinforcement" are reviewed to show the difference in their origins. Reward refers to the fact that certain environmental stimuli have the property of eliciting approach responses. Evidence suggests that the ventral striatum (nucleus accumbens area) is central to the mediation of this behavior. Reinforcement refers to the tendency of certain stimuli to strengthen learned stimulus-response tendencies. The dorsolateral striatum appears to be central to the mediation of this behavior. Neuroanatomical and neurochemical data are ...

Abstract

The concept of the ventral striatum was first put forth by Heimer and Wilson to describe the extension of basal ganglia elements into the olfactory tubercle. The ventral striatum includes the conventional nucleus accumbens, which has been closely associated with reward and motivation. This paper uses the afferent connections to the ventral striatum to define this region in monkeys. Furthermore the shell and core subterritories are discussed with respect to their histochemistry and specific connections. ...

Abstract

Previous studies have implicated the orbitofrontal cortex (OFC) in outcome encoding. However, it remains unknown whether the OFC is selectively involved in pavlovian stimulus-outcome learning or whether it also contributes to instrumental action-outcome learning. In experiment 1, we investigated this issue by assessing the effects of bilateral lesions of the OFC on the sensitivity of instrumental lever press performance to a reduction in the incentive value of the training outcome (a test of action-outcome encoding) and to outcome-specific pavlovian-instrumental transfer (a ...

Abstract

We present a model for predicting consumers' choices under conditions of uncertainty and ambiguity. We use the term ambiguity to distinguish the class of risky decisions for which the odds of an uncertain event are not precisely known. We show that our model predicts different decisions for individuals who are ambiguity averse, ambiguity seeking, or ambiguity indifferent, thus relaxing the constraint imposed on preferences by subjected expected utility theory. ...

Abstract

Economic choice is the behaviour observed when individuals select one among many available options. There is no intrinsically 'correct' answer: economic choice depends on subjective preferences. This behaviour is traditionally the object of economic analysis and is also of primary interest in psychology. However, the underlying mental processes and neuronal mechanisms are not well understood. Theories of human and animal choice have a cornerstone in the concept of 'value'. Consider, for example, a monkey offered one raisin versus one piece of ...

Abstract

Rewards are often not only valued according to their physical characteristics but also relative to other available rewards. The striatum (caudate nucleus, putamen, ventral striatum including nucleus accumbens) is involved in the organization of movement and the processing of reward information. We studied the activity of single striatal neurons in macaques that were presented with different combinations of two rewards. We found in nearly half of the investigated neurons that the processing for one reward shifted, relative to the other rewards ...

Abstract

As any child knows, the first step in counting is summing up individual elements, yet the brain mechanisms responsible for this process remain obscure. Here we show, for the first time, that a population of neurons in the lateral intraparietal area of monkeys encodes the total number of elements within their classical receptive fields in a graded fashion, across a wide range of numerical values (2-32). Moreover, modulation of neuronal activity by visual quantity developed rapidly, within 100 ms of stimulus ...

Abstract

Transient changes in the activity of midbrain dopamine neurons encode an error signal that contributes to associative learning. Although considerable attention has been devoted to the mechanisms contributing to phasic increases in dopamine activity, less is known about the origin of the transient cessation in firing accompanying the unexpected loss of a predicted reward. Recent studies suggesting that the lateral habenula (LHb) may contribute to this type of signaling in humans prompted us to evaluate the effects of LHb stimulation on ...

Abstract

The classic experiments of Yarbus over 50 years ago revealed that saccadic eye movements reflect cognitive processes. But it is only recently that three separate advances have greatly expanded our understanding of the intricate role of eye movements in cognitive function. The first is the demonstration of the pervasive role of the task in guiding where and when to fixate. The second has been the recognition of the role of internal reward in guiding eye and body movements, revealed especially in ...

Abstract

Tobacco use in developed countries is estimated to be the single largest cause of premature death. Nicotine is the primary component of tobacco that drives use, and like other addictive drugs, nicotine reinforces self-administration and place preference in animal studies. Midbrain dopamine neurons normally help to shape behaviour by reinforcing biologically rewarding events, but addictive drugs such as cocaine can inappropriately exert a reinforcing influence by acting upon the mesolimbic dopamine system. Here we show that the same concentration of nicotine ...

Abstract

Empirical research with nonhuman primates appears to support the view that causal reasoning is a key cognitive faculty that divides humans from animals. The claim is that animals approximate causal learning using associative processes. The present results cast doubt on that conclusion. Rats made causal inferences in a basic task that taps into core features of causal reasoning without requiring complex physical knowledge. They derived predictions of the outcomes of interventions after passive observational learning of different kinds of causal models. ...

Abstract

Maps of the striatum, pallidum and subthalamic nucleus were established in two macaque species (Macaca mulatta and Macaca fascicularis) in stereotaxic coordinates. The cartographic method relied on the use of intracerebral, ventricular landmarks (CA: anterior commissure and CP: posterior commissure). The basal ganglia outlines, first drawn in transverse sections perpendicular to the CA-CP plane, were reconstructed on the horizontal and midsagittal planes. Maps from several individuals were superimposed and statistical variations studied. The results confirm that the length between the two ...

Abstract

The authors present a unified account of 2 neural systems concerned with the development and expression of adaptive behaviors: a mesencephalic dopamine system for reinforcement learning and a "generic" error-processing system associated with the anterior cingulate cortex. The existence of the error-processing system has been inferred from the error-related negativity (ERN), a component of the event-related brain potential elicited when human participants commit errors in reaction-time tasks. The authors propose that the ERN is generated when a negative reinforcement learning signal ...

Abstract

The sources of input and the behavioral effects of lesions and drug administration suggest that the striatum participates in motivational processes. We investigated the activity of single striatal neurons of monkeys in response to reward delivered for performing in a go-nogo task. A drop of liquid was given each time the animal correctly executed or withheld an arm movement in reaction to a visual stimulus. Of 1593 neurons, 115 showed increased activity in response to delivery of liquid reward in both ...

Abstract

The estimation of the reward an action will yield is critical in decision-making. To elucidate the role of the basal ganglia in this process, we recorded striatal neurons of monkeys who chose between left and right handle turns, based on the estimated reward probabilities of the actions. During a delay period before the choices, the activity of more than one-third of striatal projection neurons was selective to the values of one of the two actions. Fewer neurons were tuned to relative ...

Abstract

Dopamine neurons appear to code an error in the prediction of reward. They are activated by unpredicted rewards, are not influenced by predicted rewards, and are depressed when a predicted reward is omitted. After conditioning, they respond to reward-predicting stimuli in a similar manner. With these characteristics, the dopamine response strongly resembles the predictive reinforcement teaching signal of neural network models implementing the temporal difference learning algorithm. This study explored a neural network model that used a reward-prediction error signal strongly ...

Abstract

Six adult human females chose between pairs of different amounts and delays of reinforcement scheduled according to concurrent variable-interval schedules. Consistent with previous results obtained using pigeons, the present findings were described by a version of the generalized matching law in which the relative effects of amount and delay on preference are independent and multiplicative. Therefore adult humans' greater tendency than pigeons to show self-control (i.e., adult humans' greater tendency to choose larger, more delayed over smaller, less delayed reinforcers) cannot ...

Abstract

Tuning curves are widely used to characterize the responses of sensory neurons to external stimuli, but there is an ongoing debate as to their role in sensory processing. Commonly, it is assumed that a neuron's role is to encode the stimulus at the tuning curve peak, because high firing rates are the neuron's most distinct responses. In contrast, many theoretical and empirical studies have noted that nearby stimuli are most easily discriminated in high-slope regions of the tuning curve. Here, we ...

Abstract

There have been three main ideas about the basic law of psychophysics. In 1860, Fechner used Weber's law to infer that the subjective sense of intensity is related to the physical intensity of a stimulus by a logarithmic function (the Weber-Fechner law). A hundred years later, Stevens refuted Fechner's law by showing that direct reports of subjective intensity are related to the physical intensity of stimuli by a power law. MacKay soon showed, however, that the logarithmic and power laws are ...

Abstract

The ventral tegmental area (VTA) and in particular VTA dopamine (DA) neurons are postulated to play a central role in reward, motivation and drug addiction. However, most evidence implicating VTA DA neurons in these functions is based on indirect electrophysiological characterization, rather than cytochemical identification. These physiological criteria were first established in the substantia nigra pars compacta (SNc), but their validity in the VTA is uncertain. In the current study we found that while 88 +/- 2% of SNc neurons labelled ...

Abstract

Humans and other vertebrates occasionally show a preference for items remembered to be costly or experienced when the subject was in a poor condition (this is known as a sunk-costs fallacy or state-dependent valuation). Whether these mechanisms shared across vertebrates are the result of convergence toward an adaptive solution or evolutionary relicts reflecting common ancestral traits is unknown. Here we show that state-dependent valuation also occurs in an invertebrate, the desert locust Schistocerca gregaria (Orthoptera: Acrididae). Given the latter's phylogenetic and ...

Abstract

Theories of dopamine function are at a crossroads. Computational models derived from single-unit recordings capture changes in dopaminergic neuron firing rate as a prediction error signal. These models employ the prediction error signal in two roles: learning to predict future rewarding events and biasing action choice. Conversely, pharmacological inhibition or lesion of dopaminergic neuron function diminishes the ability of an animal to motivate behaviors directed at acquiring rewards. These lesion experiments have raised the possibility that dopamine release encodes a measure ...

Abstract

There is much interest currently in using functional neuroimaging techniques to understand better the nature of cognition. One particular practice that has become common is 'reverse inference', by which the engagement of a particular cognitive process is inferred from the activation of a particular brain region. Such inferences are not deductively valid, but can still provide some information. Using a Bayesian analysis of the BrainMap neuroimaging database, I characterize the amount of additional evidence in favor of the engagement of a ...

Abstract

The use-dependent modification of synapses is strongly influenced by dopamine, a transmitter that participates in both the physiology and pathophysiology of animal behavior. In the hippocampus, dopaminergic signaling is thought to play a key role in protein synthesis-dependent forms of synaptic plasticity. The molecular mechanisms by which dopamine influences synaptic function, however, are not well understood. Using a GFP-based reporter, as well as a small-molecule reporter of endogenous protein synthesis, we show that dopamine D1/D5 receptor activation stimulates local protein synthesis ...

Abstract

A functional polymorphism in the promoter region of the human serotonin transporter gene (SLC6A4) has been associated with several dimensions of neuroticism and psychopathology, especially anxiety traits, but the predictive value of this genotype against these complex behaviors has been inconsistent. Serotonin [5- hydroxytryptamine, (5-HT)] function influences normal fear as well as pathological anxiety, behaviors critically dependent on the amygdala in animal models and in clinical studies. We now report that individuals with one or two copies of the short allele ...

Abstract

The social behavior of both human and nonhuman primates relies on specializations for the recognition of individuals, their facial expressions, and their direction of gaze [1-5]. A broad network of cortical and subcortical structures has been implicated in face processing, yet it is unclear whether co-occurring dimensions of face stimuli, such as expression and direction of gaze, are processed jointly or independently by anatomically and functionally segregated neural structures. Awake macaques were presented with a set of monkey faces displaying aggressive, ...

Abstract

In the middle temporal (MT) area of primates, many motion-sensitive neurons with a wide range of preferred directions respond to a stimulus moving in a single direction. These neurons are involved in direction perception, but it is not clear how perceptual decisions are related to the population response. We recorded the activities of MT neurons in rhesus monkeys while they discriminated closely related directions, and examined the relationship between the activities of neurons tuned to different directions and the monkeys' choices. ...

Abstract

This article surveys bootstrap methods for producing good approximate confidence intervals. The goal is to improve by an order of magnitude upon the accuracy of the standard intervals ?? ± z(?) ??, in a way that allows routine application even to very complicated problems. Both theory and examples are used to show how this is done. The first seven sections provide a heuristic overview of four bootstrap confidence interval procedures: BCa, bootstrap-t, ABC and calibration. Sections 8 and 9 describe the ...

Abstract

Food preferences are acquired through experience and can exert strong influence on choice behavior. In order to choose which food to consume, it is necessary to maintain a predictive representation of the subjective value of the associated food stimulus. Here, we explore the neural mechanisms by which such predictive representations are learned through classical conditioning. Human subjects were scanned using fMRI while learning associations between arbitrary visual stimuli and subsequent delivery of one of five different food flavors. Using a temporal ...

Abstract

In arbitrary visuomotor mapping, an object instructs a particular action or target of action, but does so in a particular way. In other forms of visuomotor control, the object is either the target of action (termed standard mapping) or its location provides the information needed for targeting (termed transformational mapping). By contrast, in arbitrary mapping, the object's location bears no systematic spatial relationship with the action. Neuropsychological and neurophysiological investigation has, in large part, identified the neural network that underlies the ...

Abstract

In addition to cells specifically active with visual stimuli, saccades, or fixation, the supplementary eye field contains cells that fire in precise temporal relationship with the occurrence of reward. We studied reward-related activity in two monkeys performing a prosaccade/antisaccade task and in one monkey trained in memory prosaccades only. Two types of neurons were distinguished by their reciprocal firing pattern: reward-predicting (RP) and reward-detecting (RD). RP neurons linearly increased their firing as early as 150 ms before saccade onset until the ...

Abstract

Mesolimbic dopaminergic neurons are thought to serve as a final common neural pathway for mediating reinforcement processes. However, several recent findings have challenged the view that mesolimbic dopamine has a crucial role in the maintenance of reinforcement processes, or the subjective rewarding actions of natural rewards and drugs of abuse. Instead, there is growing evidence that dopamine is involved in the formation of associations between salient contextual stimuli and internal rewarding or aversive events. This evidence suggests that dopaminergic-neuron activation aids ...

Abstract

Actions become compulsive when they are no longer controlled by their consequences. Compulsivity can be assessed using the omission procedure in which animals are required to withhold a previously reinforced action to earn reward. The current study tested the hypothesis that inactivation of the dorsolateral striatum (DLS), a structure implicated in habitual behavior, can enhance sensitivity to changes in the action-outcome contingency during omission training, thus leading to a reduction in compulsive responding. Over 10 days rats were trained to press ...

Abstract

Clocks tick, bridges and skyscrapers vibrate, neuronal networks oscillate. Are neuronal oscillations an inevitable by-product, similar to bridge vibrations, or an essential part of the brain's design? Mammalian cortical neurons form behavior-dependent oscillating networks of various sizes, which span five orders of magnitude in frequency. These oscillations are phylogenetically preserved, suggesting that they are functionally relevant. Recent findings indicate that network oscillations bias input selection, temporally link neurons into assemblies, and facilitate synaptic plasticity, mechanisms that cooperatively support temporal representation and ...

Abstract

BACKGROUND: The nucleus accumbens is the ventral extent of the striatum, the main input nucleus of the basal ganglia. Recent hypotheses propose that the accumbens and its dopamine projection from the midbrain contribute to appetitive behaviors required to obtain reward. However, the specific nature of this contribution is unclear. In contrast, significant advances have been made in understanding the role of the dorsal striatum in action selection and decision making. OBJECTIVE: In order to develop a hypothesis of the role of ...

Abstract

People experience regret when they realize that they would have been better off had they decided differently. Hence, a central element in regret is the comparability of a decision outcome with the outcomes forgone. Up to now, however, the comparison process that is so essential to the experience of regret has not been the subject of psychological research. In this article, we tune in on the comparison dependency of regret. We argue that factors that reduce the tendency to compare attenuate ...

Abstract

Spatiotemporal properties of dopamine release play a major role both in striatal and nigral physiology because dopamine is released from nerve terminals and dendrites of nigrostriatal dopaminergic (DA) neurons. Pioneering work revealed gap junctional communication (assessed by dye-coupling experiments) between DA cells in the substantia nigra pars compacta (SNc). However, direct evidence of functional electrical synapses between DA neurons is still lacking. In this study, gap junctional communication between DA neurons was investigated in rat brain slices. Tracer coupling was observed ...

Abstract

The authors present and test a new method of teaching Bayesian reasoning, something about which previous teaching studies reported little success. Based on G. Gigerenzer and U. Hoffrage's (1995) ecological framework, the authors wrote a computerized tutorial program to train people to construct frequency representations (representation training) rather than to insert probabilities into Bayes's rule (rule training). Bayesian computations are simpler to perform with natural frequencies than with probabilities, and there are evolutionary reasons for assuming that cognitive algorithms have been ...

Abstract

The superior colliculus (SC) is part of a network of brain areas that directs saccadic eye movements, overtly shifting both gaze and attention from position to position, in space. Here, we seek direct evidence that the SC also contributes to the control of covert spatial attention, a process that focuses attention on a region of space different from the point of gaze. While requiring monkeys to keep their gaze fixed, we tested whether microstimulation of a specific location in the SC ...

Abstract

We would like to know whether the statistics of neuronal responses vary across cortical areas. We examined stimulus-elicited spike count response distributions in V1 and inferior temporal (IT) cortices of awake monkeys. In both areas, the distribution of spike counts for each stimulus was well described by a Gaussian distribution, with the log of the variance in the spike count linearly related to the log of the mean spike count. Two significant differences in response characteristics were found: both the range ...

Abstract

Making choices is a fundamental aspect of human life. For over a century experimental economists have characterized the decisions people make based on the concept of a utility function. This function increases with increasing desirability of the outcome, and people are assumed to make decisions so as to maximize utility. When utility depends on several variables, indifference curves arise that represent outcomes with identical utility that are therefore equally desirable. Whereas in economics utility is studied in terms of goods and ...

Abstract

Single neurons can signal subtle changes in the sensory environment with surprising fidelity, often matching the perceptual sensitivity of trained psychophysical observers. This similarity poses an intriguing puzzle: why is psychophysical sensitivity not greater than that of single neurons? Pooling responses across neurons should average out noise in the activity of single cells, leading to substantially improved psychophysical performance. If, however, noise is correlated among these neurons, the beneficial effects of pooling would be diminished. To assess correlation within a pool, ...

Abstract

Concepts of motivation are vital to progress in behavioral neuroscience. Motivational concepts help us to understand what limbic brain systems are chiefly evolved to do, i.e., to mediate psychological processes that guide real behavior. This article evaluates some major motivation concepts that have historic importance or have influenced the interpretation of behavioral neuroscience research. These concepts include homeostasis, setpoints and settling points, intervening variables, hydraulic drives, drive reduction, appetitive and consummatory behavior, opponent processes, hedonic reactions, incentive motivation, drive centers, dedicated ...

Abstract

Electrical microstimulation has been used to elucidate cortical function. This review discusses neuronal excitability and effective current spread estimated by using three different methods: 1) single-cell recording, 2) behavioral methods, and 3) functional magnetic resonance imaging (fMRI). The excitability properties of the stimulated elements in neocortex obtained using these methods were found to be comparable. These properties suggested that microstimulation activates the most excitable elements in cortex, that is, by and large the fibers of the pyramidal cells. Effective current spread ...

Abstract

Behavioral phenotypes are generally complex, reflecting the action of multiple different genes. Nevertheless, there is growing evidence that key gene variants can alter activity within specific neuronal circuits and, therefore, influence particular cognitive-affective phenomena. One example is the catechol-O-methyltransferase (COMT) gene, which has a common variant at codon 158. Those with valine (Val158) alleles have increased greater COMT activity and lower prefrontal extracellular dopamine compared with those with the methionine (Met158) substitution. Val158 alleles may be associated with an advantage in ...

Abstract

It is well established that dopamine is released in the nucleus accumbens (NAC) in animals in rewarding or reinforcing situations, and widely believed that this release is the substrate of, or at least closely related to, the experience of reward. The demonstration of conditioned release of dopamine by stimuli conditioned to primary rewards has reinforced this view. However, a number of observations do not sit comfortably with this interpretation, most notably that dopamine is released equally effectively in NAC by aversive ...

Abstract

Single neurons in the ventral striatum of primates carry signals that are related to reward and motivation. When monkeys performed a task requiring one to three bar release trials to be completed successfully before a reward was given, they seemed more motivated as the rewarded trials approached; they responded more quickly and accurately. When the monkeys were cued as to the progress of the schedule, 89 out of 150 ventral striatal neurons responded in at least one part of the task: ...

Abstract

An important challenge in the design and analysis of event-related or single-trial functional magnetic resonance imaging (fMRI) experiments is to optimize statistical efficiency, i.e., the accuracy with which the event-related hemodynamic response to different stimuli can be estimated for a given amount of imaging time. Several studies have suggested that using a fixed inter-stimulus-interval (ISI) of at least 15 sec results in optimal statistical efficiency or power and that using shorter ISIs results in a severe loss of power. In contrast, ...

Abstract

The human striatum has been implicated in processing reward-related information. More recently, activity in the striatum, particularly the caudate nucleus, has been observed when a contingency between behavior and reward exists, suggesting a role for the caudate in reinforcement-based learning. Using a gambling paradigm, in which affective feedback (reward and punishment) followed simple, random guesses on a trial by trial basis, we sought to investigate the role of the caudate nucleus as reward-related learning progressed. Participants were instructed to make a ...