CiteULike: Group: Glimcher_Lab - with tag learning

Tonically active neurons in the striatum encode motivational contexts of action.

M Kimura — 2008-01-02T16:32:00-00:00

Brain Dev, Vol. 25 Suppl 1 (December 2003)

In order to achieve a goal, one procures immediately available rewards, escape from aversive events or endures absence of rewards. The neuronal substrate for these goal-directed actions includes the limbic system and the basal ganglia. In the basal ganglia, classes of projection neurons in the striatum show activity with motivational as well as sensorimotor properties, such as expectation of reward and task schedule for obtaining reward. Tonically active neurons (TANs), presumed cholinergic interneurons in the striatum, respond to reward-associated stimuli, evolve their activity through learning and respond also to aversive event-associated stimuli such as airpuff on the face. A recent study showed that responses to visual cues are less selective to whether the cue instructs reward or no reward. To address this paradox, we asked macaque monkeys to perform a set of visual reaction time tasks while expecting the reward, aversive event or absence of reward. We found that TANs respond to instruction stimuli associated with motivational outcomes but not to unassociated ones, and that they mostly differentiate associated instructions. We also found that the higher percentage of TANs in the caudate nucleus respond to stimuli associated with motivational outcomes than in the putamen, whereas the higher percentage of TANs in the putamen respond to GO signals than in the caudate nucleus especially for an action anticipating a reward. These findings suggest a distinct, pivotal role played by TANs in the caudate nucleus and putamen in encoding instructed motivational contexts for goal-directed action selection and learning in the striatum.

Reinforcement Learning: A Survey

Leslie Kaelbling — 2004-12-22T19:38:30-00:00

Journal of Artificial Intelligence Research, Vol. 4 (1996), pp. 237-285.

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of currentwork are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in...

Goal-directed instrumental action: contingency and incentive learning and their cortical substrates

Bernard Balleine — 2007-07-23T18:29:47-00:00

Neuropharmacology, Vol. 37, No. 4-5. (5 April 1998), pp. 407-419.

Instrumental behaviour is controlled by two systems: a stimulus-response habit mechanism and a goal-directed process that involves two forms of learning. The first is learning about the instrumental contingency between the response and reward, whereas the second consists of the acquisition of incentive value by the reward. Evidence for contingency learning comes from studies of reward devaluation and from demonstrations that instrumental performance is sensitive not only the probability of contiguous reward but also to the probability of unpaired rewards. The process of incentive learning is evident in the acquisition of control over performance by primary motivational states. Preliminary lesion studies of the rat suggest that the prelimibic area of prefrontal cortex plays a role in the contingency learning, whereas the incentive learning for food rewards involves the insular cortex.

A new paradigm to analyze observational learning in rats.

MG Leggio — 2007-07-13T16:11:01-00:00

Brain Res Brain Res Protoc, Vol. 12, No. 2. (October 2003), pp. 83-90.

A new paradigm of learning was developed through observational training in which rats repeatedly observed companion rats performing different spatial tasks. Observer animals were separately housed in small cages suspended over a water maze tank. They repeatedly observed companion actor rats performing spatial tasks differing according to the experimental requirements. After the observational training, observer animals were or not surgically hemicerebellectomized. This surgical ablation was performed to block any further acquisition of new behavioral strategies during actual performance of swimming task. When cerebellar symptomatology stabilized, observer animals were actually tested in the Morris Water Maze (MWM) task they had previously only observed. The observer rats displayed exploration abilities that closely matched the previously observed behaviors. The results obtained indicate that it is possible to learn complex behavioral strategies by observation using this new protocol. Furthermore, acquisition of the single facets that form the behavioral repertoire can be separately studied.

Cognitive imitation in rhesus macaques.

F Subiaul — 2007-07-13T18:55:23-00:00

Science, Vol. 305, No. 5682. (16 July 2004), pp. 407-410.

Experiments on imitation typically evaluate a student's ability to copy some feature of an expert's motor behavior. Here, we describe a type of observational learning in which a student copies a cognitive rule rather than a specific motor action. Two rhesus macaques were trained to respond, in a prescribed order, to different sets of photographs that were displayed on a touch-sensitive monitor. Because the position of the photographs varied randomly from trial to trial, sequences could not be learned by motor imitation. Both monkeys learned new sequences more rapidly after observing an expert execute those sequences than when they had to learn new sequences entirely by trial and error.

Representation of actions in rats: the role of cerebellum in learning spatial performances by observation.

MG Leggio — 2007-07-13T18:55:50-00:00

Proc Natl Acad Sci U S A, Vol. 97, No. 5. (29 February 2000), pp. 2320-2325.

Experimental evidence demonstrates that cerebellar networks are involved in spatial learning, controlling the acquisition of exploration strategies without blocking motor execution of the task. Action learning by observation has been considered somehow related to motor physiology, because it provides a way of learning performances that is almost as effective as the actual execution of actions. Neuroimaging studies demonstrate that observation of movements performed by others, imagination of actions, and actual execution of motor performances share common neural substrates and that the cerebellum is among these shared areas. The present paper analyzes the effects of observation in learning a spatial task, focusing on the cerebellar role in learning a spatial ability through observation. We allowed normal rats to observe 200 Morris water maze trials performed by companion rats. After this observation training, "observer" rats underwent a hemicerebellectomy and then were tested in the Morris water maze. In spite of the cerebellar lesion, they displayed no spatial defects, exhibiting exploration abilities comparable to controls. When the cerebellar lesion preceded observation training, a complete lack of spatial observational learning was observed. Thus, as demonstrated already for the acquisition of spatial procedures through actual execution, cerebellar circuits appear to play a key role in the acquisition of spatial procedures also through observation. In conclusion, the present results provide strong support for a common neural basis in the observation of actions that are to be reproduced as well as in the actual production of the same actions.

Observational Learning and the Feeding Behavior of the Red Squirrel Tamiasciurus Hudsonicus: The Ontogeny of Optimization

Peter Weigl — 2007-07-13T18:58:44-00:00

Ecology, Vol. 61, No. 2. (1980), pp. 213-218.

The red squirrel is a food opportunist, exploiting the seed resources in many types of forest and possessing the capacity for learning new feeding techniques. Two groups of naive, individually isolated squirrels were presented with hickory nuts for the first time. One group could observe an experienced squirrel (Model) feeding. Comparisons of the metabolic cost of feeding. times, and techniques after 6 wk showed that the group with the Model expended half the time and energy used by other group while feeding and approached the Model in time and technique. The improved performance persisted after removal of the Model. While feeding techniques may develop by trial and error, observational learning (for example, from a parent or other conspecific) is more efficient in terms of energy and time and may be crucial to the safe and rapid exploitation of new foods and habitats.

On the cognitive basis of observational learning: development of mechanisms for the detection and correction of errors.

Y Blandin — 2007-07-13T19:05:00-00:00

Q J Exp Psychol A, Vol. 53, No. 3. (August 2000), pp. 846-867.

It has been proposed that observation of a model practising a motor skill results in the observer developing mechanisms for the detection and correction of errors that are similar to those acquired during physical practice. Results of a first experiment indicated that prior observation of a model permitted participants to estimate their errors as efficiently as those who had physically practised the task. Similarly, results of a second experiment indicated that observation of a model receiving biased knowledge of results during practice resulted in similarly biased reference and error detection/correction mechanisms for the observers and for the models. These results suggest that observation engages one in cognitive processes similar to those occurring during physical practice.

Double dissociation of basolateral and central amygdala lesions on the general and outcome-specific forms of pavlovian-instrumental transfer.

LH Corbit — 2005-02-01T19:23:45-00:00

J Neurosci, Vol. 25, No. 4. (26 January 2005), pp. 962-970.

This series of experiments compared the effects of lesions of the basolateral complex (BLA) and the central nucleus (CN) of the amygdala on a number of tests of instrumental learning and performance and particularly on the contribution of these structures to the specific and general forms of pavlovian-instrumental transfer (PIT). In experiment 1, groups of BLA-, CN-, and sham-lesioned rats were first trained to press two levers, each earning a unique food outcome (pellets or sucrose), after which they were given training in which two auditory stimuli (tone and white noise) were paired with these same outcomes. Tests of specific satiety induced outcome devaluation, and tests of PIT revealed that, although the rats in all of the groups performed similarly during both the instrumental and pavlovian acquisition phases, BLA, but not CN, lesions abolished selective sensitivity to a change in the reward value of the instrumental outcome as well as to the selective excitatory effects of reward-related cues in PIT. In experiment 2, we developed a procedure in which both the general motivational and the specific excitatory effects of pavlovian cues could be assessed in the same animal and found that BLA lesions abolished the outcome-specific but spared the general motivational effects of pavlovian cues. In contrast, lesions of CN abolished the general motivational but spared the specific effects of these cues. Together, these results suggest that the BLA mediates outcome-specific incentive processes, whereas CN is involved in controlling the general motivational influence of reward-related events.

The Human Striatum is Necessary for Responding to Changes in Stimulus Relevance

R Cools — 2007-05-16T17:26:58-00:00

J. Cogn. Neurosci., Vol. 18, No. 12. (1 December 2006), pp. 1973-1983.

Various lines of evidence suggest that the striatum is implicated in cognitive flexibility. The neuropsychological evidence has, for the most part, been based on research with patients with Parkinson's disease, which is accompanied by chemical disruption of both the striatum and the prefrontal cortex. The present study examined this issue by testing patients with focal lesions of the striatum on a task measuring two forms of cognitive switching. Patients with striatal, but not frontal lobe lesions, were impaired in switching between concrete sensory stimuli. By contrast, both patient groups were unimpaired when switching between abstract task rules relative to baseline nonswitch trials. These results reveal a dissociation between two distinct forms of cognitive flexibility, providing converging evidence for a role of the striatum in flexible control functions associated with the selection of behaviorally relevant stimuli.

Machine learning approaches for phenotype-genotype mapping: predicting heterozygous mutations in the CYP21B gene from steroid profiles.

K Prank — 2007-05-03T13:19:13-00:00

Eur J Endocrinol, Vol. 153, No. 2. (August 2005), pp. 301-305.

OBJECTIVE: Non-linear relations between multiple biochemical parameters are the basis for the diagnosis of many diseases. Traditional linear analytical methods are not reliable predictors. Novel nonlinear techniques are increasingly used to improve the diagnostic accuracy of automated data interpretation. This has been exemplified in particular for the classification and diagnostic prediction of cancers based on expression profiling data. Our objective was to predict the genotype from complex biochemical data by comparing the performance of experienced clinicians to traditional linear analysis, and to novel non-linear analytical methods. DESIGN AND METHODS: As a model, we used a well-defined set of interconnected data consisting of unstimulated serum levels of steroid intermediates assessed in 54 subjects heterozygous for a mutation of the 21-hydroxylase gene (CYP21B) and in 43 healthy controls. RESULTS: The genetic alteration was predicted from the pattern of steroid levels with an accuracy of 39% by clinicians and of 64% by linear analysis. In contrast, non-linear analysis, such as self-organizing artificial neural networks, support vector machines, and nearest neighbour classifiers, allowed for higher accuracy up to 83%. CONCLUSIONS: The successful application of these non-linear adaptive methods to capture specific biochemical problems may have generalized implications for biochemical testing in many areas. Nonlinear analytical techniques such as neural networks, support vector machines, and nearest neighbour classifiers may serve as an important adjunct to the decision process of a human investigator not 'trained' in a specific complex clinical or laboratory setting and may aid them to classify the problem more directly.

Learning and selective attention.

P Dayan — 2006-03-10T19:31:44-00:00

Nat Neurosci, Vol. 3 Suppl (November 2000), pp. 1218-1223.

Selective attention involves the differential processing of different stimuli, and has widespread psychological and neural consequences. Although computational modeling should offer a powerful way of linking observable phenomena at different levels, most work has focused on the relatively narrow issue of constraints on processing resources. By contrast, we consider statistical and informational aspects of selective attention, divorced from resource constraints, which are evident in animal conditioning experiments involving uncertain predictions and unreliable stimuli. Neuromodulatory systems and limbic structures are known to underlie attentional effects in such tasks.

Learning-Related Human Brain Activations Reflecting Individual Finances

Philippe Tobler — 2007-04-06T17:24:06-00:00

Neuron, Vol. 54, No. 1. (5 April 2007), pp. 167-175.

Summary A basic tenet of microeconomics suggests that the subjective value of financial gains decreases with increasing assets of individuals ("marginal utility"). Using concepts from learning theory and microeconomics, we assessed the capacity of financial rewards to elicit behavioral and neuronal changes during reward-predictive learning in participants with different financial backgrounds. Behavioral learning speed during both acquisition and extinction correlated negatively with the assets of the participants, irrespective of education and age. Correspondingly, response changes in midbrain and striatum measured with functional magnetic resonance imaging were slower during both acquisition and extinction with increasing assets and income of the participants. By contrast, asymptotic magnitudes of behavioral and neuronal responses after learning were unrelated to personal finances. The inverse relationship of behavioral and neuronal learning speed with personal finances is compatible with the general concept of decreasing marginal utility with increasing wealth.

Little Information, Efficiency and Learning - An Experimental Study

Atanasios Mitropoulos — 2007-04-23T14:16:43-00:00

Earlier experiments have shown that under little information subjects are hardly able to coordinate even though there are no conflicting interests and subjects are organised in fixed pairs. This is so, even though a simple adjustment process would lead the subjects into the efficient, fair and individually payoff maximising outcome. We draw on this finding and design an experiment in which subjects repeatedly play 4 simple games within 4 sets of 40 rounds under little information. This way...

Information Theory, Inference and Learning Algorithms

D Mckay — 2007-04-17T18:15:39-00:00

Learning in the presence of malicious errors

Michael Kearns — 2007-04-17T18:05:01-00:00

(1988), pp. 267-280.

In this paper we study an extension of the distribution-free model of learning introduced by Valiant [23] (also known as the probably approximately correct or PAC model) that allows the presence of malicious errors in the examples given to a learning algorithm. Such errors are generated by an adversary with unbounded computational power and access to the entire history of the learning algorithm's computation. Thus, we study a worst-case model of errors.

FrontalTemporal Disconnection Abolishes Object Discrimination Learning Set in Macaque Monkeys

Browning — 2007-03-24T18:30:39-00:00

Cerebral Cortex, Vol. 17, No. 4. (April 2007), pp. 859-864.

The roles of the caudate nucleus in human classification learning.

CA Seger — 2007-02-27T03:35:05-00:00

J Neurosci, Vol. 25, No. 11. (16 March 2005), pp. 2941-2951.

The caudate nucleus is commonly active when learning relationships between stimuli and responses or categories. Previous research has not differentiated between the contributions to learning in the caudate and its contributions to executive functions such as feedback processing. We used event-related functional magnetic resonance imaging while participants learned to categorize visual stimuli as predicting "rain" or "sun." In each trial, participants viewed a stimulus, indicated their prediction via a button press, and then received feedback. Conditions were defined on the bases of stimulus-outcome contingency (deterministic, probabilistic, and random) and feedback (negative and positive). A region of interest analysis was used to examine activity in the head of the caudate, body/tail of the caudate, and putamen. Activity associated with successful learning was localized in the body and tail of the caudate and putamen; this activity increased as the stimulus-outcome contingencies were learned. In contrast, activity in the head of the caudate and ventral striatum was associated most strongly with processing feedback and decreased across trials. The left superior frontal gyrus was more active for deterministic than probabilistic stimuli; conversely, extrastriate visual areas were more active for probabilistic than deterministic stimuli. Overall, hippocampal activity was associated with receiving positive feedback but not with correct classification. Successful learning correlated positively with activity in the body and tail of the caudate nucleus and negatively with activity in the hippocampus.

Dynamics of frontal, striatal, and hippocampal systems during rule learning.

CA Seger — 2007-02-27T03:34:14-00:00

Cereb Cortex, Vol. 16, No. 11. (November 2006), pp. 1546-1555.

We examined interactions between frontal, striatal, and hippocampal systems during a rule-learning task. Nineteen healthy young adults solved multiple rule-learning problems requiring hypothesis testing while functional magnetic resonance images were obtained. Activity in the head of the caudate peaked early after the beginning of each problem and then dropped rapidly. In contrast, activity in prefrontal cortex areas reached peak values later. These results are in accordance with theories suggesting that the striatum identifies the behavioral context necessary for the frontal lobe to select an appropriate strategy. Striatal and hippocampal systems showed antagonistic patterns of activity: Activation in the anterior hippocampus decreased, whereas caudate activity increased. Good learners showed higher activity in the body and tail of the caudate than poor learners, whereas learning success correlated negatively with activity in the hippocampus. Activation in the head of the caudate correlated negatively with hippocampal activation, indicating a potential mechanism for hippocampal activity reduction.

Dissociation between Striatal Regions while Learning to Categorize via Feedback and via Observation.

CM Cincotta — 2007-02-27T03:12:26-00:00

J Cogn Neurosci, Vol. 19, No. 2. (February 2007), pp. 249-265.

Convergent evidence from functional imaging and from neuropsychological studies of basal ganglia disorders indicates that the striatum is involved in learning to categorize visual stimuli with feedback. However, it is unclear which cognitive process or processes involved in categorization is or are responsible for striatal recruitment; different regions of the striatum have been linked to feedback processing and to acquisition of stimulus-category associations. We examined the effect of the presence of feedback during learning on striatal recruitment by comparing feedback learning with observational learning of an information integration task. In the feedback task, participants were shown a stimulus, made a button press response, and then received feedback as to whether they had made the correct response. In the observational task, participants were given the category label before the stimulus appeared and then made a button press indicating the correct category membership. A region-of-interest analysis was used to examine activity in three regions of the striatum: the head of the caudate, body and tail of the caudate, and the putamen. Activity in the left head of the caudate was modulated by the presence of feedback: The magnitude of activation change was greater during feedback learning than during observational learning. In contrast, the bilateral body and tail of the caudate and the putamen were active to a similar degree in both feedback and observational learning. This pattern of results supports a functional dissociation between regions of the striatum, such that the head of the caudate is involved in feedback processing, whereas the body and tail of the caudate and the putamen are involved in learning stimulus-category associations. The hippocampus was active bilaterally during both feedback and observational learning, indicating potential parallel involvement with the striatum in information integration category learning.

Reward or reinforcement: what's the difference?

NM White — 2007-02-23T20:17:02-00:00

Neurosci Biobehav Rev, Vol. 13, No. 2-3. (l 1989), pp. 181-186.

The histories of the terms "reward" and "reinforcement" are reviewed to show the difference in their origins. Reward refers to the fact that certain environmental stimuli have the property of eliciting approach responses. Evidence suggests that the ventral striatum (nucleus accumbens area) is central to the mediation of this behavior. Reinforcement refers to the tendency of certain stimuli to strengthen learned stimulus-response tendencies. The dorsolateral striatum appears to be central to the mediation of this behavior. Neuroanatomical and neurochemical data are adduced suggesting that reward may be mediated by a neural circuit including the neostriatal patch system, together with the hippocampus, limbic system (amygdala, prefrontal cortex) and ventral pallidum. The evidence also suggests that reinforcement, in the form of dopamine release in the striatal matrix, acts to promote the consolidation of sensori-motor associations. Thus, the matrix may mediate stimulus-response memory as part of a circuit including the cerebral cortex, substantia nigra pars reticulata and its projections to thalamic and brainstem motor areas.

Statistical Learning Within and Between Modalities: Pitting Abstract Against Stimulus-Specific Representations

Conway — 2006-11-16T05:19:28-00:00

Psychological Science, Vol. 17, No. 10. (October 2006), pp. 905-912.

Habit and Skill Learning in Schizophrenia: Evidence of Normal Striatal Processing With Abnormal Cortical Input

Thomas Weickert — 2007-02-13T23:55:34-00:00

Learn. Mem., Vol. 9, No. 6. (1 November 2002), pp. 430-442.

Different forms of nondeclarative learning involve regionally specific striatal circuits. The motor circuit (involving the putamen) has been associated with motor-skill learning and the dorsolateral prefrontal cortex (DLPFC) circuit (involving the caudate) has been associated with cognitive-habit learning. Efforts to differentiate functional striatal circuits within patient samples have been limited. Previous studies have provided mixed results regarding striatal-dependent nondeclarative learning deficits in patients with schizophrenia. In this study, a cognitive-habit learning task (probabilistic weather prediction) was used to assess the DLPFC circuit and a motor-skill learning task (pursuit rotor) was used to assess the motor circuit in 35 patients with schizophrenia and 35 normal controls. Patients with schizophrenia displayed significant performance differences from controls on both nondeclarative tasks; however, cognitive-habit learning rate in patients did not differ from controls. There were performance and learning-rate differences on the motor-skill learning task between the whole sample of patients and controls, however, analysis of a subset of patients and controls matched on general intellectual level eliminated learning rate differences between groups. The abnormal performance offset between patients with schizophrenia and controls in the absence of learning rate differences suggests that abnormal cortical processing provides altered input to normal striatal circuitry. 10.1101/lm.49102

A survey on the bandit problem with switching costs

Tackseung Jun — 2007-01-04T05:10:58-00:00

De Economist, Vol. V152, No. 4. (1 December 2004), pp. 513-541.

Bees in two-armed bandit situations: foraging choices and possible decision mechanisms

Tamar Keasar — 2007-01-04T02:58:43-00:00

Behav. Ecol., Vol. 13, No. 6. (1 November 2002), pp. 757-765.

In multi-armed bandit situations, gamblers must choose repeatedly between options that differ in reward probability, without prior information on the options' relative profitability. Foraging bumblebees encounter similar situations when choosing repeatedly among flower species that differ in food rewards. Unlike proficient gamblers, bumblebees do not choose the highest-rewarding option exclusively. This incomplete exclusiveness may reflect an adaptive sampling strategy. A cost--benefit analysis predicts decreased sampling levels with increasing differences in mean profitability between the available food sources. We simulated two-armed bandit situations in laboratory experiments to test this prediction. Bumblebees (Bombus terrestris L.) made 300 foraging visits to blue and yellow artificial flowers that dispensed sucrose solution according to seven probabilistic reward schedules. Reward schedules varied in profitability differences between the two feeding options. As predicted, the bees specialized more on the higher-rewarding food type (and thus sampled the alternative less) when the mean reward difference between the feeding options was larger. Choice ratios of individual bees were linearly related to the reward ratios they had experienced. It has been suggested that the behavioral mechanism underlying incomplete exclusiveness may involve simple rules of thumb that do not require long-term memory. However, the bees' response to recent foraging experience (rewarded and non-rewarded visits) differed between the beginning and the end of observation sessions and between treatments. Simulations of the Rescorla-Wagner difference learning rule reproduced the main trends of the results. These findings suggest that the observed incomplete exclusiveness results from associative learning involving long-term memory. 10.1093/beheco/13.6.757

Selective enhancement of associative learning by microstimulation of the anterior caudate

Ziv Williams — 2006-03-27T16:11:32-00:00

Nature Neuroscience, Vol. 9, No. 4. (26 February 2006), pp. 562-568.

Modelling the acquisition of goal-directed behaviors by populations of neurons.

E Guigon — 2006-09-27T22:46:39-00:00

Int J Psychophysiol, Vol. 19, No. 2. (March 1995), pp. 103-113.

Recent neurophysiological studies have revealed the patterns of neuronal activity during the acquisition of goal-directed behaviors, both in single cells, and in large populations of neurons. We propose a model which helps three sets of experimental results in the monkey to be understood: (1) activity of single cells vary greatly and only population activities are causally related to behavior. The model shows how a population of stochastic neurons, whose behaviors vary widely, can learn a skilled conditioned movement with only local activity-dependent synaptic changes. (2) typical changes in neuronal activity occur when the rules governing the behavior are changed, i.e. when the relationship between cues and actions to reach a goal changes over time. There are two types of neuronal patterns during changes in reward contingency: a monotonic increasing pattern and a non-monotonic pattern which follows the change in the way the reward is obtained. Units in the model display these two types of change, which correspond to synaptic modifications related to the encoding of the behavioral significance of sensory and motor events. (3) These two patterns of neuronal activity define two populations whose anatomical distributions in the frontal lobe overlap with a gradient organized in the rostro-caudal direction. The model consists of two artificial neural networks, defined by the same set of equations, but which differ in the values of two parameters (P and Q). P defines the adaptive properties of processing units and Q describes the coding of information. The model suggests that a balance in the relative strengths of these parameters distributed along a rostro-caudal gradient can explain the distribution of neuronal types in the frontal lobe of the monkey.

The basal ganglia in human learning.

CA Seger — 2006-08-16T21:58:22-00:00

Neuroscientist, Vol. 12, No. 4. (August 2006), pp. 285-290.

For many years, the basal ganglia were described in anatomy courses as strictly motor structures. Certainly, some of the most obvious and debilitating symptoms shown by persons with basal ganglia disorders are problems in motor control. However, the basal ganglia are not limited to motoric aspects of behavior: recent research shows that they are involved in most areas of cognitive and emotional functioning, consistent with their anatomical connections with all areas of the cortex. This review will focus on the roles of the basal ganglia in human learning, particularly sequence learning and category learning. Current areas of research that are discussed include the differing roles of different basal ganglia regions, patterns of interaction between the cortex and basal ganglia, differences in positive and negative association learning, effects of dopaminergic medication on learning, whether basal ganglia-mediated learning is implicit or explicit, and how the basal ganglia learning systems interact with other learning systems, particularly within the medial temporal lobe.

Early consolidation of instrumental learning requires protein synthesis in the nucleus accumbens

Pepe Hernandez — 2006-05-30T04:03:33-00:00

Nat Neurosci, Vol. 5, No. 12. (December 2002), pp. 1327-1331.

The primate amygdala represents the positive and negative value of visual stimuli during learning

Joseph Paton — 2006-02-15T19:40:49-00:00

Nature, Vol. 439, No. 7078. (16 February 2006), pp. 865-870.

General Gittins index processes in discrete time.

N El Karoui — 2006-06-08T21:02:39-00:00

Proc Natl Acad Sci U S A, Vol. 90, No. 4. (15 February 1993), pp. 1232-1236.

We combine the formulation of Mandelbaum [Mandelbaum, A. (1986) Probab. Theory Rel. Fields 71, 129-147] with ideas from Whittle [Whittle, P. (1980) J. R. Stat. Soc. B 42, 143-149] to obtain a simple and constructive proof for the optimality of Gittins index processes in the general, nonmarkovian dynamic allocation (or "multi-armed bandit") problem. Our approach also provides an explicit expression for the value of this problem.

Generalization in vision and motor control

Tomaso Poggio — 2005-10-18T09:10:30-00:00

Nature, Vol. 431, No. 7010. (14 October 2004), pp. 768-774.

Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria

Ido Erev — 2005-11-22T17:57:12-00:00

The American Economic Review, Vol. 88, No. 4. (1998), pp. 848-881.

We examine learning in all experiments we could locate involving 100 periods or more of games with a unique equilibrium in mixed strategies, and in a new experiment. We study both the ex post ("best fit") descriptive power of learning models, and their ex ante predictive power, by simulating each experiment using parameters estimated from the other experiments. Even a one-parameter reinforcement learning model robustly outperforms the equilibrium predictions. Predictive power is improved by adding "forgetting" and "experimentation," or by allowing greater rationality as in probabilistic fictitious play. Implications for developing a low-rationality, cognitive game theory are discussed.

What songbirds teach us about learning.

MS Brainard — 2006-04-17T15:36:01-00:00

Nature, Vol. 417, No. 6886. (16 May 2002), pp. 351-358.

Bird fanciers have known for centuries that songbirds learn their songs. This learning has striking parallels to speech acquisition: like humans, birds must hear the sounds of adults during a sensitive period, and must hear their own voice while learning to vocalize. With the discovery and investigation of discrete brain structures required for singing, songbirds are now providing insights into neural mechanisms of learning. Aided by a wealth of behavioural observations and species diversity, studies in songbirds are addressing such basic issues in neuroscience as perceptual and sensorimotor learning, developmental regulation of plasticity, and the control and function of adult neurogenesis.

A Neural Substrate of Prediction and Reward

Wolfram Schultz — 2005-07-29T17:08:25-00:00

Science, Vol. 275, No. 5306. (14 March 1997), pp. 1593-1599.

Instructive signals for motor learning from visual cortical area MT

Megan Carey — 2005-05-25T19:50:13-00:00

Nature Neuroscience, Vol. 8, No. 6. (22 May 2005), pp. 813-819.

Dopamine responses comply with basic assumptions of formal learning theory.

P Waelti — 2005-04-08T21:45:00-00:00

Nature, Vol. 412, No. 6842. (5 July 2001), pp. 43-48.

According to contemporary learning theories, the discrepancy, or error, between the actual and predicted reward determines whether learning occurs when a stimulus is paired with a reward. The role of prediction errors is directly demonstrated by the observation that learning is blocked when the stimulus is paired with a fully predicted reward. By using this blocking procedure, we show that the responses of dopamine neurons to conditioned stimuli was governed differentially by the occurrence of reward prediction errors rather than stimulus-reward associations alone, as was the learning of behavioural reactions. Both behavioural and neuronal learning occurred predominantly when dopamine neurons registered a reward prediction error at the time of the reward. Our data indicate that the use of analytical tests derived from formal behavioural learning theory provides a powerful approach for studying the role of single neurons in learning.

Dopaminergic Stimulation of Local Protein Synthesis Enhances Surface Expression of GluR1 and Synaptic Transmission in Hippocampal Neurons.

WB Smith — 2005-03-17T16:01:35-00:00

Neuron, Vol. 45, No. 5. (3 March 2005), pp. 765-779.

The use-dependent modification of synapses is strongly influenced by dopamine, a transmitter that participates in both the physiology and pathophysiology of animal behavior. In the hippocampus, dopaminergic signaling is thought to play a key role in protein synthesis-dependent forms of synaptic plasticity. The molecular mechanisms by which dopamine influences synaptic function, however, are not well understood. Using a GFP-based reporter, as well as a small-molecule reporter of endogenous protein synthesis, we show that dopamine D1/D5 receptor activation stimulates local protein synthesis in the dendrites of hippocampal neurons. We also identify the GluR1 subunit of AMPA receptors as one protein upregulated by dopamine receptor activation, with increased incorporation of surface GluR1 at synaptic sites. The insertion of new GluRs is accompanied by an increase in the frequency of miniature synaptic events. Together, these data suggest a local protein synthesis-dependent activation of previously silent synapses as a result of dopamine receptor stimulation.

A computational model of the functional role of the ventral-striatal D2 receptor in the expression of previously acquired behaviors.

AJ Smith — 2005-03-16T15:50:59-00:00

Neural Comput, Vol. 17, No. 2. (February 2005), pp. 361-395.

The functional role of dopamine has attracted a great deal of interest ever since it was empirically discovered that dopamine-blocking drugs could be used to treat psychosis. Specifically, the D2 receptor and its expression in the ventral striatum have emerged as pivotal in our understanding of the complex role of the neuromodulator in schizophrenia, reward, and motivation. Our departure from the ubiquitous temporal difference (TD) model of dopamine neuron firing allows us to account for a range of experimental evidence suggesting that ventral striatal dopamine D2 receptor manipulation selectively modulates motivated behavior for distal versus proximal outcomes. Whether an internal model or the TD approach (or a mixture) is better suited to a comprehensive exposition of tonic and phasic dopamine will have important implications for our understanding of reward, motivation, schizophrenia, and impulsivity. We also use the model to help unite some of the leading cognitive hypotheses of dopamine function under a computational umbrella. We have used the model ourselves to stimulate and focus new rounds of experimental research.

Dopamine: the salient issue.

MA Ungless — 2005-03-14T18:35:22-00:00

Trends Neurosci, Vol. 27, No. 12. (December 2004), pp. 702-706.

There is general agreement that midbrain dopamine neurons play key roles in reward processing. What is more controversial is the role they play in processing salient stimuli that are not rewarding. This controversy has arisen for three main reasons. First, salient sensory stimuli such as tones and lights, which are assumed not to be rewarding, increase dopamine neuron activity. Second, aversive stimuli increase firing in a minority of putative dopamine neurons. Third, dopamine release is increased following aversive stimuli. Consequently, it has been suggested that these midbrain dopamine neurons are activated by all salient stimuli, rather than specifically by rewards. However, reconsideration of these issues, in light of new findings, suggests this controversy can be resolved in favour of reward theories.

Distinguishing Whether Dopamine Regulates Liking, Wanting, and/or Learning About Rewards.

S Robinson — 2005-03-14T18:23:04-00:00

Behav Neurosci, Vol. 119, No. 1. (February 2005), pp. 5-15.

To determine whether dopamine regulates liking, wanting, and/or learning about rewards during goal-directed behavior, the authors tested genetically engineered dopamine-deficient (DD) mice for acquisition of an appetitive T-maze task with and without endogenous dopamine signaling. Experiment 1 established that DD mice treated with L-dihydroxyphenylalanine (L-dopa [LD]) perform similarly to controls on a T-maze task designed to measure liking, wanting, and learning about rewards. Experiment 2, which tested saline-, caffeine-, and LD-treated DD mice on the T maze, separated performance factors from cognitive processes and revealed that dopamine is not necessary for mice to like or learn about rewards but is necessary for mice to seek (want) rewards during goal-directed behavior. ((c) 2005 APA, all rights reserved).

Espresso reward learning, hold the dopamine: theoretical comment on robinson et Al. (2005).

KC Berridge — 2005-03-14T18:22:15-00:00

Behav Neurosci, Vol. 119, No. 1. (February 2005), pp. 336-341.

Question: Is dopamine needed for reward learning? Answer: No--at least, not in the brain of a caffeinated dopamine-deficient (DD) mutant mouse. That is the conclusion of an important paper in this issue by S. Robinson, S. M. Sandstrom, V. H. Denenberg, and R. D. Palmiter (see record 2005-01705-001). Those authors demonstrate that reward learning can proceed normally in the brains of DD mice, even though they contain no dopamine at the time of learning, if the mice are given caffeine just before learning. Caffeine activates the DD mice by a nondopaminergic mechanism, allowing them to learn where to obtain food reward in a T-maze runway. Their reward-learning-without-dopamine is revealed on a subsequent test day, when dopamine function is restored by L-dopa administration. Robinson et al. conclude that dopamine is not needed for normal learning about rewards, or for hedonic "liking" of rewards during learning, but rather specifically for a motivational "wanting" component of reward, such as incentive salience. ((c) 2005 APA, all rights reserved).

Experience-Weighted Attraction Learning in Games: A Unifying Approach

C Camerer — 2005-03-09T20:05:44-00:00

(1997)

We describe a general model, 'experience-weighted attraction' (EWA) learning, which includes reinforcement learning and a class of weighted fictitious play belief models as special cases. In EWA, strategies have attractions which reflect prior predispositions, are updated based on payoff experience, and determine choice probabilities according to some rule (e.g., logit). A key feature is a parameter δ which weights the strength of hypothetical reinforcement of strategies which were not chosen according to the payoff they would have yielded. When δ = 0 choice reinforcement results. When δ = 1, levels of reinforcement of strategies are proportional to expected payoffs given beliefs based on past history. Another key feature is the growth rates of attractions. The EWA model controls the growth rates by two decay parameters, φ and ρ, which depreciate attractions and amount of experience separately. When φ = ρ belief-based models result; when ρ = 0 choice reinforcement results. Using three data sets, parameter estimates of the model were calibrated on part of the data and used to predict the rest. Estimates of δ are generally around .50, φ around 1, and ρ varies from 0 to φ. Choice reinforcement models often outperform belief-based models in the calibration phase and underperform in out-of-sample validation. Both special cases are generally rejected in favor of EWA, though sometimes belief models do better. EWA is able to combine the best features of both approaches, allowing attractions to begin and grow exibly as choice reinforcement does, but reinforcing unchosen strategies substantially as belief-based models implicitly do. http://ideas.repec.org/p/clt/sswopa/1003.html

Different time courses of learning-related activity in the prefrontal cortex and striatum

Anitha Pasupathy — 2005-02-23T20:24:42-00:00

Nature, Vol. 433, No. 7028. (24 February 2005), pp. 873-876.

Experience-dependent, asymmetric expansion of hippocampal place fields.

MR Mehta — 2005-02-18T16:57:40-00:00

Proc Natl Acad Sci U S A, Vol. 94, No. 16. (5 August 1997), pp. 8918-8921.

Theories of sequence learning based on temporally asymmetric, Hebbian long-term potentiation predict that during route learning the spatial firing distributions of hippocampal neurons should enlarge in a direction opposite to the animal's movement. On a route AB, increased synaptic drive from cells representing A would cause cells representing B to fire earlier and more robustly. These effects appeared within a few laps in rats running on closed tracks. This provides indirect evidence for Hebbian synaptic plasticity and a functional explanation for why place cells become directionally selective during route following, namely, to preserve the synaptic asymmetry necessary to encode the sequence direction.

Experience-dependent asymmetric shape of hippocampal receptive fields.

MR Mehta — 2005-02-18T16:55:37-00:00

Neuron, Vol. 25, No. 3. (March 2000), pp. 707-715.

We propose a novel parameter, namely, the skewness, or asymmetry, of the shape of a receptive field to characterize two properties of hippocampal place fields. First, a majority of hippocampal receptive fields on linear tracks are negatively skewed, such that during a single pass the firing rate is low as the rat enters the field but high as it exits. Second, while the place fields are symmetric at the beginning of a session, they become highly asymmetric with experience. Further experiments suggest that these results are likely to arise due to synaptic plasticity during behavior. Using a purely feed forward neural network model, we show that following repeated directional activation, NMDA-dependent long-term potentiation/long-term depotentiation (LTP/LTD) could result in an experience-dependent asymmetrization of receptive fields.

Neural activity in the primate prefrontal cortex during associative learning.

WF Asaad — 2005-02-18T15:48:12-00:00

Neuron, Vol. 21, No. 6. (December 1998), pp. 1399-1407.

The prefrontal (PF) cortex has been implicated in the remarkable ability of primates to form and rearrange arbitrary associations rapidly. This ability was studied in two monkeys, using a task that required them to learn to make specific saccades in response to particular cues and then repeatedly reverse these responses. We found that the activity of individual PF neurons represented both the cues and the associated responses, perhaps providing a neural substrate for their association. Furthermore, during learning, neural activity conveyed the direction of the animals' impending responses progressively earlier within each successive trial. The final level of activity just before the response, however, was unaffected by learning. These results suggest a role for the PF cortex in learning arbitrary cue-response associations, an ability critical for complex behavior.

The representation of time for motor learning.

JF Medina — 2005-02-08T20:16:49-00:00

Neuron, Vol. 45, No. 1. (6 January 2005), pp. 157-167.

We have identified factors that control precise motor timing by studying learning in smooth pursuit eye movements. Monkeys tracked a target that moved horizontally for a fixed time interval before changing direction through the addition of a vertical component of motion. After repeated presentations of the same target trajectory, infrequent probe trials of purely horizontal target motion evoked a vertical eye movement around the time when the change in target direction would have occurred. The pursuit system timed the vertical eye movement by keeping track of the duration of horizontal target motion and by measuring the distance the target traveled before changing direction, but not by learning the position in space where the target changed direction. We conclude that high temporal precision in motor output relies on multiple signals whose contributions to timing vary according to task requirements.