Dopamine Neurons: Reward, Aversion, or Both?

May 27 2009 Published by under Neuroscience

If you can't tell by now, Sci is something of a dopamine junkie.

Anyway, when one first learns about dopamine, you learn about a "reward" molecule, the one that makes you feel good. Sounds like dope for a reason. But over time, scientist have found that it's not just about reward with dopamine. Dopamine has a lot more to do with things we like to call salience and value. The salience of a cue is in part related to its strength, and it part related to what its connected with. Basically, a cue is high salience if it gives you a good reason to pay attention. It isn't attention itself, it's being connected to something worth paying attention to. This is connected to the item's value. After all, if it's something I don't value, the article isn't going to be very salient to me, it won't be worth paying attention to.
So as of recently, it's been assumed that dopamine neurons fire in response to value-related signals. Sci's dopamine neurons fire in response to pizza, and a crack addict's neurons fire in response to cocaine. And of course, if they encode value-related stimuli, dopamine neurons should be inhibited by aversive stimuli, because those have negative value. So while my dopamine neurons fire in response to pizza, they should be really inhibited in response to brussels sprouts.

Right? Well...wrong. And this is something that has puzzled scientists for a while. Some studies show inhibition of dopamine neurons in response to negative stimuli, and some show both negative and POSITIVE dopamine response to negative stimuli. So what's up with that? Are the neurons firing for negative stimuli just some random wackos that get off on brussels sprouts?
Well, it's possible that they aren't wackos. It's possible that the dopamine neurons really do just encode value-relation. Not whether that value is positive or negative. Scientists recently have tried to test this, and what they found clarifies a lot of things we didn't know about the firing of dopamine neurons.
So what do you need to make a groundbreaking paper (in this case a paper in Nature)? Some juice, some air, and a couple of monkeys. Matsumoto and Hikosaka. "Two types of dopamine neuron distinctly convey positive and negative motivational signals" Nature, 2009.

Let me just say here. One of the things I love about papers in Nature and Science is the fact they they are limited with regard to space. I know from experience, however, that writing a paper when you're limited on space is teh suxxors. Only allowed 40 references?! WHAT?! How can they expect me to place this paper in the context of the literature correctly if I only get 40 references?!
But really, some of the space limitations (NOT the references, in my opinion), are a good thing. When you only have 8 columns of space (including figures) to write intro, results, and discussion, you've got to be CLEAR. You have to be CONCISE. You gotta tell your story and tell it straight. And what this often results crap. Completely illegible crap where scientists feel they have to use torturous expressions and excessive use of long words to convey something that is often relatively basic. Tell it to me straight. Please. Pretend I'm stupid and don't know what you're talking about. Chances are, you're right.
But though the result is often crap, it's not always. And this is what I love about this paper. It's short, and it's SWEET. Clear and concisely written. I applaud these guys (or girls, they could be girls, unfortunately, I am not well versed enough in Japanese to know whether "Masayuki" and "Okihide" are male or female names). They have the courage and talent to do what many scientists cannot: write a straight story.
And here it is:
Take two monkeys. Implant a TON of electrodes into the substantia nigra and the ventral tegmental area, and capture a whole bunch of neurons (103 to be exact). Make sure they are dopamine neurons (you can tell this by the electrophysiological trace, which will be different from that of the surrounding neurons, most of which are GABA cell bodies).
Then, once the monkeys are recovered from surgery, start training them on a simple task. A Pavlovian response. Pavlov, for those not versed in this class experiment, trained dogs to expect treats at the sound of a bell. After a while, when the bell was rung, the dogs would start salivating massively in expectation of their food. This is a phenomenon now known as classical conditioning, and one that many mammals do very well. Pair a stimulus (like a light or tone) with a condition (like food or something aversive, like a bad taste or smell). Pretty soon, the animal (or human) will react to the stimulus alone, even if the condition doesn't materialize.
So in this case, with the monkeys, they used positive and negative conditions. One set of signals indicated a certain probability of a juice reward. A spot meant 100% chance of juice. A cross meant 50% change of juice, or no juice. A square meant you got nothing. Like this:

Then they had a second set of conditions, this time connected with a puff of air in the eyes. Nothing painful, just the kind of thing to make you blink and get you irritated. In this condition, an open circle meant 100% chance of air puff, a triangle meant a 50% chance of an air puff or nothing, and a square meant nothing. Like this:

Not surprisingly, the animals' behavior soon corresponded closely with the signals it was getting. If it saw the spot and knew it was getting juice, licking at the spout would increase in anticipation. If it saw the open circle and knew it was getting an air puff, it would blink to try and avoid the air puff.
Ok fine, Pavlovian conditioning. But remember those electrodes in the brain? Those electrodes attached to...dopamine neurons? Well, it turns out that all the neurons responded to stimuli, but they broke down into two distinct categories when it came to WHICH stimuli they responded to. The first set were clearly responding to the positive reward, the juice:

You can see there, ,starting in the upper right, the dopamine cells spiking in response to stimuli. Because they are trained to the reward and the stimulus, the monkeys are no longer responding to the reward itself, but rather to the stimulus, which, in the case of juice, will trigger behaviors like licking at the juice spout in anticipation. On the left, you can see the spike when the monkeys the the spot, 100% chance of juice showers, let the licking begin! In the center panel, there is still a spike, but it's a much smaller one, there's only a 50% chance of juice, and so their brains are weighing the chances. In the far right, you can see the sign for no juice. The neurons aren't just quiet or baseline, they're inhibited, no chance of reward here.
And this first set of neurons responded in the opposite manner to the chance of an airpuff:

On the left we have 100% chance of an air puff, and totally inhibited neuronal activity. Clearly, no chance for juice. On the right, however, there is the 0% air puff stimulus, and the neurons fire. *phew* escaped the air puff to the face!
But then, there was a SECOND set of neurons. In the reward paradigm, they look just the same as the first set. But in the airpuff paradigm, you see something different:

These neurons, as you can see on the left side, respond to a certain air puff, which is known to be no kind of rewarding stimulus, are are actually inhibited when there is no chance of an air puff. But they respond to juice in a normal manner. WTF?
So they did the experiment again, only this time, they recorded neurons when the animals got an unexpected freebie, either free juice, or free air puff. This time the neurons organized themselves into two groups again (though slightly different groups), and the same thing happened:

Here you can see that first set of neurons. When you're just looking at juice presentation, and NOT at the stimulus, the neurons don't fire (on the top left) even though there's 100% chance of reward. Instead, they fire in response to the stimulus, as you saw in the first figure. But on the top right, you can see the strong response when they just get some free juice. When it's not connected with a stimulus, that juice will still make those neurons fire all day long. In contrast, on the bottom right, when the monkeys get a free air puff, the neurons go silent. Stimulus or not, those particular dopamine neurons do not appreciate that air puff.
But what about the second set?

Same thing as in the first paradigm, they still respond well to a free reward (top right), but they ALSO respond to a free air puff.
What does this mean? It means that though there is a set of dopamine neurons that responds to value-related stimuli (the juice only), there is ALSO a set of neurons that just responds to salience, something you need to be paying attention to, and thus they would respond to both the juice AND the air puff.
But the coolest part was yet to come. Not only were the researchers able to identify two distinct populations of neurons, those neurons were grouped clearly in different places!

The dots you can see in red are the ones that responded to both the juice and the air puff, while the ones in the blue responded only to the juice. And a pattern emerges. The red neurons are much more dorsolaterally located, ending up in the substantia nigra, while the blue ones which appear to code for value only were in the ventral tegmental area.
This has some big implications. First of all, it shows conclusively that not all dopamine neurons will respond only to positive stimuli and show inhibition to negative stimuli, some will respond to both. Secondly, it shows that these neurons can be localized to distinct populations, which give some clues as to their function. The substantia nigra pars compacta is an area of dopamine cell bodies that is known to degenerate in diseases like Parkinson's, while the ventral tegmental area is related to the mesoaccumbens dopamine circuit thought to be involved in things like motivation and addictions. Thus, it makes more sense for the value-only related neurons to be localized to the ventral tegmental area, while the others, which appear to encode salience rather than value AND salience, are localized to the substantia nigra, which is known to have a myriad of functions.
So why is this important? Understanding how we pay attention to things and encode things like value responding in the brain can play a big role in how we target treatments for things like addiction, which dramatically changes the firing activity of some of these neurons. Not only that, it could help find treatments for other disorders of motivation and salience, particularly things like Obsessive-Compulsive disorder and kleptomania. It's amazing what you can do with two monkeys, some juice, and some air.
Matsumoto, M., & Hikosaka, O. (2009). Two types of dopamine neuron distinctly convey positive and negative motivational signals Nature DOI: 10.1038/nature08028

15 responses so far

  • Angela says:

    Scicurious, great post.
    I was wondering if you could say a little more about the different parts of the brain? I am not a neuro person and was a bit lost when you mentioned:
    "neurons are much more dorsolaterally located, ending up in the substantia nigra, while the blue ones which appear to code for value only were in the ventral tegmental area"
    Also, what is the mesoaccumbens dopamine circuit?

  • DrJohn says:

    Schultz has been advocating DA as a prediction-error signal since at least 2001, and most of the data we have backs it up. Even cooler, most of what he's seen (and us too) suggests that there is in fact little to no functional differentiation between VTA and SNc. Freaky stuff...

  • Scicurious says:

    Angela: Sorry about that, I was thinking as I wrote that I probably ought to say something on that topic. I've got a couple of posts on basic neuroanatomy (here, here, and here) that may be able to give you a good idea.
    But for the specifics of this particular paper:
    dorsolateral refers to something being higher up in the brain and further out to the side (as opposed to near the middle). The substantia nigra pars compacta and the ventral tegmental area are both the main areas where dopamine neurons are located in the brain. The substantia nigra is most well known for the role we think it plays in Parkinson's disease, where dopamine neurons degrade and intentional movements are strongly affected (though, of course, it isn't that simple). The ventral tegmental area, which is directly adjacent to the substantia nigra, also contains dopamine neurons, though these project more toward regions like the nucleus accumbens, an area associated with things like the initial rewarding and reinforcing effects of drugs.
    The ventral tegmental area is ALSO part of the mesoaccumbens dopamine circuit, a set of dopamine neuron projections running from the ventral tegmental area to the accumbens, and projecting out and connecting to higher centers. It's also known as the reward circuit, though of course we know this is way too simple of a way to think about it.
    Does that help? Let me know if you have any more questions, I can see if I can provide some pics.
    DrJohn: No function difference between VTA and SNc? I can't say I'm surprised...but it kind of quirks my brain a little...

  • Zen Faulkes says:

    Nice post. Please excuse a slightly picky tangent...
    You imply molecules determine the response of the postsynaptic neurons in the paragraph starting, "when one first learns about dopamine, you learn about a 'reward' molecule, the one that makes you feel good," which suggests that molecule itself has some built in properties that neurons simply interpret. Then you switch to "dopamine neurons," which is better, but it doesn't quite get rid of the implication that the molecule itself determines how the postsynaptic neurons respond.
    The effect of a neuroactive chemical depends on the postsynaptic neuron, not the molecule. A chemical can be a fast acting neurotransmitter to one postsynaptic neuron, and a slow acting neuromodulator to another. Sometimes, the same chemical can have two different effects simultaneously in the same postsynaptic neuron (Katz & Frost. 1995. J Neurophysiol 74: 2881.

  • Scicurious says:

    Good point, Zen, I didn't realize that I was sounding like that. I'll be more careful.

  • Terry says:

    are you sure you can tell which neurons are DA?

  • tero says:

    nice posting!
    i remember this mechanism been studied with similar test already in the beginning of the decade.
    basic methodology was the same, monkey in monkey chair
    receiving visual stimuli and corresponding reward/no reward. electrodes placed in SNpc an VTA. (sorry, can remember exactly) 🙁
    same dopaminergic cell responded to stimulus which was promising reward, but also slightly when there was no promise of reward.
    interestingly, these cells responded quite vigorously when there was 50/50 chance.
    this was explained as evolutionary mechanism providing reward for risk-seeking behaviour, reward for exploring for example new territory or feeding grounds. dopaminergic reward system would in this scenario reward animal for taking risks in uncertain situations.
    at least to me this seems plausible. this could be also reason for addictive behaviour with drugs or gambling. you never know what your getting but still your brain rewards you for trying something new..
    i can´t remember the authors of the study, which i am sorry for. maybe i´ll find it later an post the link.

  • Angela says:

    Thanks for that extra explanation and links scicurious, it does help 😉

  • Janne says:

    It's a good paper and it fits neatly with how the amygdala (that gets intrinsic value from the VTA, and outputs a stimulus evaluation to the SNc) and striatum work with saliency and value.
    There is a little detail about the AUS-excited neurons in fig. 3e in the paper (your next to last figure). They seem to respond to air puffs even when the puff is 100% predictable; if the SNc-based DA neurons encode saliency they should stay silent just as they do for the 100% reward case. Of course, the significance seems low and the monkeys are trained for both contingencies so it's possible we're seeing some confusion across context.

  • DDeden says:

    I may be all wrong, am not a neuroscientist, but...
    "So while my dopamine neurons fire in response to pizza"
    Nu-uh. They fire in alarm response to instabilization of homeostasis, as a protective mechanism. Can't explain it.

  • Marsh says:

    This is a request from a non-scientist and relatively old geezer.Could one of you smart, young whipper-snappers see if you can relate the awesome science in that paper to the intuition I have developed during my years of working with babies/toddlers on the autism spectrum who never develop language?
    My gut has always told me that the "nonverbal" part of "nonverbal autism" was the result of poorly regulated (prepare for scientific term that I invented) "brain happy juice." The brain happy juice in babies and toddlers with severe (usually meaning nonverbal) autism hijacks their attention and makes little addicts out of them before critical language acquiring skills get a fair shot at wiring their brains for language.
    I can't get these little guys to be interested in the stuff I need them to be interested in because this "brain happy juice" has not only hijacked their attention, but then it seems like another mechanism takes over and makes that which they are paying attention to waaaay more interesting than whatever it is that I have to offer them. And not just more interesting--but waaaay more interesting than it should be--and waaaay more interesting than anything else in the universe is to any other person in the universe!(Look!Wheels!They spin!!!)Ten years later--spinning wheels are still doing it for these little guys--and have done so at the expense of learning really basic functional skills--like human language.
    And I've always suspected some "dope dealer" in these little guys' brains is dealing the "happy brain juice."I feel like I am competing with a heroin dealer for a heroin addict's attention. And Dude! I can't compete!
    So if you guys could come up with a pill or something that makes that heroin dealer go away--even for a couple hours a day for the 6 months to three year old "at risk for nonverbal autism" crowd--I'm in! I know my way around the language machine and can start it up--all I need is a distraction so that I can sneak past the heroin dealer.
    So there you have it--your challenge for the day. Go forth, mine this research, and FIX NONVERBAL AUTISM for me. I'll give you til end of business today...

  • Pteryxx says:

    Hey Marsh - I hear autistic kids respond very well to operant conditioning, but fail at learn-by-imitation which is how most kids learn language. Also, that the presence of another person is stressful and distracting (certainly true in my case). How about pairing up spinny things as a reward for language experimentation, say in a computer program? Or use a toy fan as a reward, as we'd use a treat clicker for a dog or dolphin?
    Don't know that I can try and relate *this* paper, because if dopamine levels affect both salience and reward, then weakening attention hijacking could weaken reward feedback at the same time, which won't help learning... at least until we find medications that target one set of dopaminergic neurons over the other.
    Good luck though.

  • Marsh says:

    Pteryxx- And you don't think you can take care of that whole "targeting one set of dopaminergic neurons over the other" thing by close of business today? Because I think you have hit the nail on the head--and I think that's what I'm going to need to get in to wire up some of these language machines. How about I give you scientist-types an extension on the assignment--maybe we'll talk again in a couple of years?
    As far as operant conditioning is concerned--heck, I try it all, but, you can't get very far with it if the language system isn't wired up beforehand. Operant conditioning with my little nonverbal folks tends to result in just what you said--the trained dolphin outcome. "Touch blue." "Touch blue." "Good touching blue!" Don't get me wrong--it's a definite plan B--but we really want the first few "hits" of intensive language intervention to be warmer and fuzzier and waaaaay funner. Your ideas, by the way, are exactly on target for these little guys once--or if-- the magic begins to happen.
    And it's difficult to appreciate the "magic" that is expressive and receptive human language...until you see it not happen.
    Well--to all of you neuroscientists and neurobloggers out there...thank you for always, ALWAYS providing me with something to be hopeful about.
    p.s. note to any neuro researchers out there--If I were you, I would definitely exhaust the autism research funding entities before approaching the Kleptomania research funding entities.(Do we really know where the Kleptomania funding comes from? I mean, think about it.)

  • Pteryxx says:

    Well, because current dopaminergic medications have such complicated and annoying side effects, some of them probably ARE targeting some populations of neurons more than others, we just can't or haven't been able to distinguish which ones or why. Now we know there are (at least) two populations involved in motivational signals: how are they different? Why are they different? Are they using the exact same receptors, same cascade pathways, are they modulated by feedback from other neurons? There's a good ten years' worth of work there.
    It would be nice to test current ADHD or schizophrenia medications, but they have many side effects, the effects vary greatly by individual, and they may even have long-term developmental consequences. That calls for a very cautious, yet long-term and large-scale study to distinguish any population variances... another ten year project, at least.
    Now if I may get back to your discussion of operant conditioning... and let me say, as it comes across poorly in my text ; ) That I'm very glad to not only hear from, but make contact with, someone as warm and enthusiastic and dedicated to the challenge of teaching autistic children as you obviously are. Because I am an "aspie", mildly face-blind and deaf to emotional cues myself, so I have both a personal viewpoint and a personal interest in autistic minds. I feel that I deeply appreciate what "magic" human communication can be, because like a good magic trick, it's largely incomprehensible to me.
    Because of that viewpoint, I respectfully suggest that your description of conditioning, warm and human though it is, might not be very effective with autistic minds. Human presence, hands hugging us, faces staring and smiling at us, and loud cheerful voices saying "Good job!" are NOT pleasant stimuli - they're overwhelming, distracting, and shut down our minds. One of the reasons autistics fixate on mechanical stimuli is to protect ourselves from being drowned in social input. Temple Grandin in her books says that she wanted to be hugged as a child, but found the experience so intense and overwhelming that she could only tolerate it for a few seconds at best.
    That's why I suggested that something pretty and nonhuman be offered as a positive reinforcer, such as a few spins of a toy fan, the feel of a stuffed animal, or whatever the particular kid is drawn to. A voice saying over and over "Touch blue! Touch blue!" isn't encouragement, but annoyance to someone like me, and a loud enthusiastic "Good touch blue!" isn't a reward, it's a punishment. I'm concerned that when you say "warmer and fuzzier" interaction, you mean lots of touching and hugging and face time. That's wonderful for you, and for typical human kids, but I suggest it's actually working against your autistic kids.
    It's known that autistics score much higher on intelligence tests that do NOT require social interaction with a human test administrator. Perhaps they'll find it easier to learn language in the first place if the social interaction aspect is minimized: say, if their teacher sits behind or beside, avoids eye contact, and speaks in a calm, quiet voice. And if they're given rewards that actually feel good, such as spinny things, or silence. Social interaction isn't instinctive for us; so it needs to be learned also, but separately. It's asking a lot of these kids to learn both language and socialness at the same time.
    Also see this Wired article: The Truth About Autism
    If you're near Texas I'd be glad to meet you. I'm pteryxx at gmail dot com.
    Good luck - Pteryxx

  • DrL, your correspondent from Japan says:

    "I applaud these guys (or girls, they could be girls, unfortunately, I am not well versed enough in Japanese to know whether "Masayuki" and "Okihide" are male or female names)."
    If they are Japanese scientists from any discipline apart from Women Studies then there is 96% chance that they are boys. Even if they work in US.
    I have checked the names. Yes, indeed they are.

Leave a Reply