plospcbiplcbPLoS Comput BiolploscompPLoS Computational Biology1553-734X1553-7358Public Library of ScienceSan Francisco, USA10.1371/journal.pcbi.004000407-PLCB-RA-0549R2plcb-04-01-03Research ArticleNeurological DisordersNeuroscienceNoneSerotonin, Inhibition, and Negative MoodSerotonin, Inhibition, and Negative
MoodDayanPeter1HuysQuentin J. M12* Gatsby Computational Neuroscience Unit, University College London,
London, United Kingdom Center for Theoretical Neuroscience, Columbia University, New York, New
York, United States of America FristonKarl JEditorUniversity College London, United Kingdom* To whom correspondence should be addressed. E-mail: qhuys@cantab.net
PD and QJMH conceived and designed the experiments, performed the
experiments, analyzed the data, and wrote the paper.
The authors have declared that no competing interests exist.
220081220082711200742e41392007271120072008 Dayan and HuysThis is an open-access article distributed under
the terms of the Creative Commons Attribution License, which permits unrestricted
use, distribution, and reproduction in any medium, provided the original author and
source are credited.
Pavlovian predictions of future aversive outcomes lead to behavioral inhibition,
suppression, and withdrawal. There is considerable evidence for the involvement
of serotonin in both the learning of these predictions and the inhibitory
consequences that ensue, although less for a causal relationship between the
two. In the context of a highly simplified model of chains of affectively
charged thoughts, we interpret the combined effects of serotonin in terms of
pruning a tree of possible decisions, (i.e., eliminating those choices that have
low or negative expected outcomes). We show how a drop in behavioral inhibition,
putatively resulting from an experimentally or psychiatrically influenced drop
in serotonin, could result in unexpectedly large negative prediction errors and
a significant aversive shift in reinforcement statistics. We suggest an
interpretation of this finding that helps dissolve the apparent contradiction
between the fact that inhibition of serotonin reuptake is the first-line
treatment of depression, although serotonin itself is most strongly linked with
aversive rather than appetitive outcomes and predictions.
Author Summary
Serotonin is an evolutionarily ancient neuromodulator probably best known for
its role in psychiatric disorders. However, that role has long appeared
contradictory to its role in normal function, and indeed its various roles
in normal affective behaviors have been hard to reconcile. Here, we model
two predominant functions of normal serotonin function in a highly
simplified reinforcement learning model and show how these may explain some
of its complex roles in depression and anxiety.
Funded by the Gatsby Charitable Foundation (PD, QJMH) and a Bogue Research
Fellowship (QJMH).citationDayan P, Huys QJM (2008) Serotonin, inhibition, and negative mood.
PLoS Comput Biol 4(2): e4. doi:10.1371/journal.pcbi.0040004Introduction
Serotonin (5-hydroxytryptamine [5-HT]) is a neuromodulator that
appears to play a critical role in a wealth of psychiatric conditions, including
depression, anxiety, panic, and obsessive compulsions. However, despite the
importance of serotonergic pharmacotherapies, notably selective serotonin reuptake
inhibitors (SSRIs), the roles that serotonin plays in normal and abnormal function
are still mysterious. We start from three particular findings. First, 5-HT is
involved in the prediction of aversive events, possibly as a form of opponent
[1–3] to dopamine
[4–11]. Second, 5-HT is
involved in behavioral inhibition [12–14], preventing or curtailing ongoing
actions in light of predictions of aversive outcomes. The third finding is the
collection of psychopharmacological data implicating 5-HT in animal models of
depression and anxiety [15–17], together with the fact that depleting 5-HT (by dietary
depletion of its precursor, tryptophan) in human subjects who have recovered from
depression, can reinstate an acute, at times fulminant, re-experience of subjective
symptoms of the disease, as assessed by various rating scales [18–21]. Furthermore, while
SSRIs are used in the treatment of depression, genetically induced, constitutive
decreases in the efficiency of 5-HT reuptake are a risk factor for depression
[22–24]. These findings are hard to connect: the second fact seems
orthogonal to the first and third, which are themselves in apparent contradiction.
If 5-HT is really involved in predicting aversive outcomes, then
depleting it should surely have positive rather than negative
affective consequences.
We suggest that the missing link comes from considering the interactions between
Pavlovian predictions and ongoing action selection. The interaction is seen in
conditioned suppression [25], a standard workhorse test for aversive predictions. Animals
are trained to emit appetitive instrumental actions (such as pressing a lever for
reward), and to associate (by classical conditioning) a light with a shock.
Presentation of the light during instrumental performance reduces
the rate at which animals emit those responses. Neither the theoretical nor the
neurobiological status of this interaction is completely resolved, though there is
some evidence of the involvement of 5-HT in the nucleus accumbens in its realization
[26–28].
Here, we treat a subset of the inhibitory processes associated with Gray's behavioral
inhibition system (BIS) [7,13,29,30] in terms of what might be called a
preparatory Pavlovian response. Consummatory Pavlovian responses are
(evolutionarily) pre-programmed reactions to the presence of affectively significant
outcomes such as food, water, or threats. Preparatory Pavlovian responses are
similarly pre-programmed responses to predictions of those outcomes. Even though the
predictions are learned, the responses are not, and may therefore be behaviorally
inappropriate in certain circumstances [31,32]. For our purposes, and as long noted by Deakin and Graeff
[7], the
most important preparatory Pavlovian response to a prediction of a (sufficiently
distant) threat [30] is inhibition, in the form of withdrawal or disengagement. This
explicitly links the first two findings discussed above, as the inhibition is
directly associated with aversive predictions.
To explore the consequences of reflexive, direct inhibition of action for learning in
affective settings, together with the repercussions when 5-HT is compromised, we
built a highly simplified model that sought to isolate these effects from more
general learning effects. More specifically, we built a model of trains of thoughts.
In our treatment, we considered thoughts as actions that lead from one belief state
to the next. Trains of thought gained value through their connections with a group
of terminal states that were preassigned either positive or negative affective
values. 5-HT directly inhibited chains of thought predicted to lead toward negative
terminal states. Our model can be seen in terms of 5-HT's pruning of a decision tree
of outcome states and choices [33,34].
We argue that the results on tryptophan depletion (TrD) above now emerge when
considering the consequences of this reflexive behavioral inhibition on ongoing
learning about the world, and on subsequent action choice and predictions. The most
notable effect in the model is a critical bias toward optimistic
valuation. That is, states and actions with potentially negative consequences are
under-explored and incorrectly (over)-valued because of the reflexive inhibition.
When inhibition fails, though, which is the last of the three issues mentioned
above, there are two adverse consequences. First, the inhibition is no longer a
crutch for instrumental action choice, so subjects have to learn to avoid
potentially bad situations rather than being able to rely on this reflexive
mechanism. Second, due to a mismatch between policy and value function,
characteristic inconsistencies between the predicted and actual values arise, with
the actual values encountered being more negative than predicted, though also
actually more realistic. This mismatch between policy and value function also leads
to an overall reduction in rewards obtained. Boosting 5-HT in the model again
restores the status quo. Of course, this highly simplified model cannot possibly, by
itself, accommodate all the diverse and confusing roles of 5-HT. Nevertheless, it
replicates some prominent behavioral and pharmacological facets of depression and
anxiety in humans and animal models, which we return to in the Discussion.
The next section defines the model of trains of thought more formally. The Results
section considers normal (hence biased) learning, and the consequences of
impairments to 5-HT processing. We save for the Discussion a broader discussion of
data and theories pertaining to 5-HT.
MethodsThe Model: Trains of Thought
Figure 1 illustrates our
underlying model of trains of thought. It is intended to emphasize a role for
5-HT in behavioral inhibition, and is therefore couched at an abstract level.
Throughout, we will equate thoughts with actions, and revisit the more general
action setting later. We initially focus on the effect of one inhibitory
reflexive action in the context of otherwise fixed actions (a fixed policy).
10.1371/journal.pcbi.0040004.g001
Markov Models of Thought
The abstract state space is divided into the four blocks shown. The right
two, and , are associated with direct affective values
r(s) (inset histograms); the left
two, and , are internal. Transitions between (belief) states
are determined by actions (thoughts). We initially focus on a fixed
policy, leading to the transition between states shown in the figure:
states in each internal block and preferentially connect with each other and their
respective outcome states and . However, each state has links to states in the other
block. The model is approximately balanced as a whole, with an equal
number of positive and negative states.
A train of thoughts starts at one of a set of internal belief states
( , ), may proceed through more such states, and ends in one of set
of terminal outcome states (, ). The connectivity between belief states is sparse, with
states leading preferentially to other states in and outcome states with positive values; and states leading preferentially to other states in and outcome states with negative values (red arrows), though each could also lead
to states of opposite “sign” (black arrows in Figure 1). In addition, trains
of thought can be inhibited by 5-HT (see below). In this simple model, the value
of an internal state is the average value of the terminal states to which it
ultimately leads.
More formally, the model is a form of Markov decision process (see
[35]), with four sets of sparsely interconnected states
(, ). Two sets, and (each with 100 elements in the simulation) are associated
respectively with positive (r(s) ≥ 0,
s ∈ ) and negative affective values
(r(s) ≤ 0, s
∈ ); both are drawn from suitably truncated 0-mean, unit
variance, Gaussian distributions (see inset histograms in Figure 1) and are terminal states. The other
sets, and (each with 400 elements), contain internal states and are not
associated with immediate affective values
(r(s) = 0, ∀s
∈ ∪ ).
Serotonergic Inhibition
A policy is a (probabilistic) mapping from states to actions a
← π(s) and defines the transition matrix
between the states in the model. For simplicity, we consider a fixed, basic,
policy π0. In this, each element of effectively has eight outgoing connections: three to other
(randomly chosen) elements in ; three to randomly chosen elements in ; and one each to randomly chosen elements in and . Similarly, each element of has eight outgoing connections: three to other (randomly
chosen) elements in ; three to randomly chosen elements in ; and one each to randomly chosen elements in and . Thoughts are modelled as actions a following
these connections, labelled by the identities of the states to which they lead.
Text S1 gives details of a more complex environment in which we explicitly
explore effects of impulsivity.
To isolate the effect of 5-HT in inhibiting actions in aversive situations, we
consider the highly simplified proposal that serotonin stochastically terminates
trains of thoughts when these reach aversive states. More specifically, under
serotonergic influence the transition probabilities are modified in a manner
that depends on states' values. We let the probability of continuing a train of
thought (of continuing along the fixed policy π0) be dependent
(and inversely related to) the value V(s) of a
state: where α5HT is a multiplicative factor that
scales the impact of 5-HT (see Figure 2). When thoughts are not continued (inhibited), they stop
and restart in a randomly chosen state (though see below for relaxations of this). The more
disastrous the potential sequelæ of state s, the more
negative Vπ(s), and so the
less likely the chain was to be continued. On the other hand, even slightly
positive values would essentially veto any termination. This introduces an
asymmetry into the model defined by the simple base policy. Other possibilities
for the information reported by 5-HT and for the dynamic interaction between
5-HT and dopamine are considered in the discussion, and the fixed base policy
π0 is relaxed below.
10.1371/journal.pcbi.0040004.g002
Probability of Continuing a Train of Thoughts
For values V(s) > 0, thoughts
are continued with probability 1. Conversely, when the state
s has negative value, the probability of continuation
drops of as an exponential function of the value. The rate of the
exponential is set by α5HT.
Learning
The value of each state represents the expected reward obtainable from that state
when following a particular policy. Under the fixed policy π0,
dynamic programming techniques [35] allow the value function
Vπ(s) over states
s to be written, and solved for, concisely as:
Vπ(s) =
r(s), s ∈ , and where γ is a discount factor
(γ = 0.9 in our simulations). Dynamic
programming also uses a
function [36] over states and thoughts defined
for those actions that exist by
Optimal values V*(s) and
are those value functions associated with any policy
that maximizes the long-run affective outcomes of the train.
While it is not possible to use these techniques directly to evaluate the value
function under serotonergic influence (the inhibition depends on the value
function itself and thus represents a nonlinear interaction), the temporal
difference learning rule [35] can be used to acquire estimates
of the values
of states under serotonergically modified policies
.
The temporal difference learning rule specifies an online learning rule for
which the change in the estimated value
based on taking action a at state s and
therefore arriving deterministically at s′
= a(s) is: where the learning rate ε = 0.05. A slightly
simpler alternative rule suggests that learning of
is itself prevented by termination:
That
does not change under this rule given termination implies that learning is only
slowed for these states, rather than being biased toward zero. We generally
report results from this variant.
In the sequel, we show values after substantial learning (20,000 trains), plus
the consequences of manipulating serotonin (by manipulating
α5HT) once the values are already acquired.
Manipulations After LearningTryptophan depletion
Given the values
learned under a policy π(α5HT) determined by
α5HT = 20, the steady-state transitions
probabilities can be calculated for any new α5HT
≠ 20 simply by working out the probability of inhibition for each
state. In particular, this allows α5HT to be reduced to
model a pharmacological or psychiatric reduction in serotonin function. To
separate the effect of this reduction from that of learning, we only learn
up to the reduction and then look at the behavior after the reduction in the
absence of further learning.
Recall bias and reward seeking
To account for the effect of recall biases often seen in depression, we will
additionally consider the effect of biased resampling after behavioral
inhibition. A simple way of achieving this in a manner that relates to the
affective value of states is to let whereby values of β < 0 will bias resampling
toward states with lower values
V(s) (i.e., states in ).
So far, only serotonergically determined inhibitory responses have been
considered. Mirror to these are dopaminergically controlled approach
responses [32], which actually favour actions a with
positive state-action values (under a policy). The combined effect can be incorporated
in a straightforward manner by choosing action a in state
s according to a softmax where θ controls the degree of influence of the
value. Note that, in this simple model, instrumental and
Pavlovian control are essentially indistinguishable.
ResultsBehavioral Inhibition
By construction, the environment in Figure 1 is symmetric with respect to rewards and punishments, and
so the overall statistics of the values of states are balanced about zero.
Indeed, Figure 3A shows that
for the base policy, 20,000 learning steps are ample to acquire a reasonable
value Vest(s) for the states (the
remaining discrepancies from
Vtrue(s), here defined for
α5HT = 0, arise from the stochasticity in
the choice of action together with the fixed learning rate). Critically, there
is no bias in either Vest(s) or
Vtrue(s).
10.1371/journal.pcbi.0040004.g003
Learning with Behavioral Inhibition
(A,B) With α5HT = 0, for one particular
learning run, the values Vest match their
true values Vtrue (inferred through dynamic
programming) under an equal-sampling exploration policy (A), and trains
of thought end in terminal states , equally often as a function of their actual outcomes
(B) (the red line is the regression line).
(C,D) With α5HT = 20, negative
V values are poorly estimated (since exploration is
progressively inhibited for larger α5HT), and the
more negative the value of the outcome, the less frequently that outcome
gets visited over learning (D). Importantly, there is an optimistic
underestimate of the negative value of state.
(E) The root mean squared error (averaging over 20 runs) for states with
positive (dotted) and negative Vtrue values
as a function of α5HT. The effect of the sampling
bias is strikingly apparent, preventing accurate estimates mainly of the
negatively valued states.
(F) Average reward received during learning as a function of
α5HT—the benefits of behavioral
inhibition are apparent.
By contrast, Figure 3D shows
the substantial bias in
consequent on setting a large value of α5HT =
20. In this case, low-valued states are much less well visited and explored. The
bias comes despite the use of learning rule [5], which only slows down learning
for low-value states rather than also distorting it. Of course, in this case,
the extent of the bias depends on the initial values for the states (all of
which are set to zero in the simulation).
Figure 3E shows how
frequently each of the outcome states was reached in a run (as a function of its
outcome r(s)). Since behavioral inhibition
terminates trains on their way to potential disaster, aversive terminal states
are sampled less (shown by the red regression line), which is consistent with
the bias of the estimated value. Figures 3C and 3F show these effects as a function of α5HT. The
greater the inhibition, the worse estimated the values are (Figure 3C), particularly for aversive states;
however, the more benign is the exploration (Figure 3F). Learning with greater inhibition
leads to a more optimistic set of values; however, this is coupled with a more
aggressive rejection of all actions even mildly associated with negative
outcomes.
Tryptophan Depletion
Reducing the value of α5HT after learning a value function
under its influence can be expected to have various consequences, as it
introduces a mismatch between policy and value function. The most obvious one is
a more negative average affective outcome (the average value of trains of
thought) in the model. This is because choices are less biased against actions
that are predicted to have aversive consequences, and so the latter occur more
frequently. A second consequence is that there will be substantial adverse
surprises associated with transitions that previously were inhibited. The
surprise at reaching an actual outcome can be measured using the prediction
error for the last transition of a chain from state
to a state
.
We may expect negative prediction errors to be of special importance, because of substantial evidence that
aversive outcomes whose magnitudes and timing are expected so they can be
prepared for, have substantially less disutility than outcomes that are more
aversive than expected (at least for physiological pains; see [37]).
Figure 4 shows the
consequences of learning under full inhibition and then wandering through state
space with reduced inhibition. The change in the average terminal affective
value as a percentage of the case during learning that α5HT
= 20 is shown in Figure
4A. As was already apparent in Figure 3F (which averages over the whole
course of learning), large costs are incurred for large reductions in
inhibition. For α5HT = 0, the average reward is
actually negative, which is why the curve dips below −100%.
This value is relevant, since the internal environment is approximately
symmetric in terms of the appetitive and aversive outcomes it affords. Subjects
normally experience an optimistic or rosy view of it, by
terminating any unfortunate trains of thought (indeed, 55% of their
state occupancy is in compared with ). Under reduced 5-HT, subjects see it more the way it really
is (the ratio becomes 50%).
10.1371/journal.pcbi.0040004.g004
Reduced Inhibition
These graphs show statistics of the effect of learning V
values with α5HT = 20, and then
suffering from reduced serotonin α5HT < 20
during sampling of thoughts. For a given thought environment, these are
calculated in closed form, without estimation error.
(A) As is also evident in Figure 3F, the average affective return is greatly reduced
from the value with α5HT = 20; in fact,
for the extreme value of α5HT = 0, it
becomes slightly negative (reflecting a small sample bias in the
particular collection of outcomes).
(B,C) Normalized outcome prediction errors at the time
of transition to (B) or (C) for α5HT = 20
against α5HT = 0. These reflect the
individual probability that each terminal transition goes to
r(s) from
V(s′) for
s ∈ and
s′ ∈ , including all the probabilistic contingencies of
termination, etc. They are normalized for the two values of
α5HT. Terminations in are largely unaffected by the change in inhibition;
terminations in with negative consequences have greatly increased
negative prediction error.
Figure 4B and 4C show comparative scatter
plots of the terminal prediction errors. Here, we consider just the last
transition from an internal state to an outcome state. Prediction errors here
that are large and negative, with substantially more aversive outcomes than
expected, may be particularly damaging. Figure 4C compares the average terminal
prediction errors for all transitions into states in with no serotonergic inhibition α5HT
= 0, to those for the value α5HT = 20
that were used during learning. For the case that α5HT
= 20, the negative prediction errors are on average very small
(partly since the probability of receiving one is very low). With reduced
inhibition, the errors become dramatically larger, potentially leading to
enhanced global aversion. By comparison, as one might expect, the positive
prediction errors resulting from transitions into are not greatly affected by the inhibition (Figure 4B).
Recall Bias
Two additional effects enrich this partial picture. One, which plays a
particularly important role in the cognitive behavioral therapy literature, is
that depressed patients have a tendency to prefer to recall
aversive states or memories [38,39]. Figure 5A shows the consequence of doing this
according to a simple softmax (see Methods). These curves, as in Figure 4A, show the percentage average
utility compared with α5HT = 20, β
= 0 across values of α5HT, and for β
= −10, −9,…,10. As might be expected,
biasing the starting point to , and, even worse, to those particular states in
that are most deleterious, has a big negative impact on
average utility. For α5HT = 0; β
= −10, occupancy of relative to became a paltry 27% as subjects ruminate
[40,41] negatively.
10.1371/journal.pcbi.0040004.g005
Reward Seeking and Recall Bias
Both plots are in the same form as Figure 4A, showing the percentage
utilities compared with the standard learning case
α5HT = 20, as a function of
α5HT (the emboldened blue curve is exactly that
in Figure 4A).
(A) Given a mood-dependent bias on the starting state, with
,
the plots show the consequences of various values of β.
Negative β, favoring low value states, leads to substantially
negative average outcomes.
(B) Instrumental control of action choice, a putative model of
dopaminergic effects, can also either exacerbate or improve the
outcomes, depending on the value of the parameter θ governing
a softmax choice of actions.
Reward Seeking
The second factor is our restriction to just inhibition of trains of thought
rather than a more fine-scale manipulation of the relative probabilities of
different thoughts. We now relax this and explore the effect of additionally
allowing preferential transitions toward certain states. In Equation 6, for
positive values, the parameter θ biases action choice toward actions
leading to positively valued states, whereas for negative values it does the
opposite (i.e., subjects prefer to transition to negatively valued states).
Figure 5B shows the
effects of θ. It is apparent that rather extreme values of θ
can both significantly aggravate or suppress the effect of
α5HT. For the highest positive values of θ the
curves reverse shape, showing that it can be beneficial not to
inhibit trains of thought. This arises since the model of Figure 1 was chosen to have the extreme
property that there is always the possibility of avoidance (in that all the
states in admit at least one action that leads to ), and inhibiting trains of thought removes this outcome. A
different, and rather counterintuitive, interaction between inhibition and
reward seeking obtains in environments where rewards are hidden behind
punishments (see Text S1 and Figure
S1).
Discussion
We studied a very simple Markov decision process model of affectively charged
thoughts, and showed various aspects of the influence of behavioral inhibition on
the experience of appetitive and aversive outcomes, predictions, and prediction
errors. The model formalises behavioral inhibition as a Pavlovian control process
that arrests internally directed thoughts (and likewise externally directed actions)
that are predicted to lead to aversive consequences. Overall this is favourable,
leads to enhanced average rewards, and is related to adaptive pruning
[33,34]. However, the
consequences can also be deleterious [31,32]. Compromising inhibition in the model has two related
consequences. First, the values of states are revealed to be overly optimistic.
Second, control is disturbed, with aversive chains being insufficiently deselected.
While this work shows how several prominent aspects of serotonin's manifold putative
functions and effects can be reconciled within a unifying framework, we acknowledge
that we have neglected a wide range of other issues, and certainly do not claim that
this is an exhaustive account of the data. There is also an interesting alternative
view of 5-HT, such as that due to [42] who suggested that it is involved in
controlling the appropriate timescale of behavior by determining the discount factor
for future affective outcomes (parameter γ in Equation 2). In this
theory, 5-HT depletion reduces the effective value of γ,
making subjects appear more impulsive [43–45]. Our model captures impulsivity
through reduced 5-HT more directly, suggesting that actions that are comparatively
worse lose direct inhibition that was previously restraining them, and are therefore
more likely to be executed.
Behavioral Inhibition System
We suggested that this form of behavioral inhibition arises through predictions
of aversive outcomes, tied to serotonin's putative role in reporting aversive
prediction errors as an opponent to dopamine. This comes directly from the
original notions of behavioral inhibition and serotonergic effects from Gray,
Deakin, Graeff, and their colleagues [6,7,13,29,30]; however, it is perhaps best
seen as a subset of the current version of Gray's BIS [29]. One salient
difference is that BIS is suggested as being primarily engaged by
conflict, rather than ongoing predictions of future
aversive outcomes. Of course, a main source of conflict is that between approach
and avoidance, with the latter coming from these aversive predictions. An
interesting consequence of dividing the prediction of the value of future
outcomes between two separate opponent systems is that it is indeed possible to
have simultaneous appetitive and aversive expectations, as opposed to just one
combined net prediction. Although we used the net prediction to control
inhibition, it would be interesting to explore other possibilities associated
with the BIS view, such as that any aversive prediction could arrest ongoing
action, even if outweighed by appetitive predictions.
Further, rather than have the aversive predictive values of
states lead to termination of trains of thought, it is possible that the
negative prediction error (δ− from Equation 8), which
Daw et al. [10] suggested is being reported by phasic serotonin, could be
responsible instead. Alternatively, in the mirror reflection of the proposal
that a tonic dopaminergic signal reports average reward (and
controllable/avoidable punishment) and energises behavior [46,47], it could be that a more tonic
serotonergic signal, averaging aversion over longer time horizons and favoring
quiescence, could be responsible.
Another difference between our account and the full BIS is that, in the latter,
although actions are indeed inhibited in the face of conflict, the BIS is then
suggested as initiating a set of behaviors (such as exploration or risk
assessment) to resolve that conflict. The set of preparatory Pavlovian actions
associated with aversive predictions appears to be more refined than that
associated with appetitive predictions (mostly just approach), with a wide range
of different defensive possibilities being selected between according to the
nature and proximity of the threat [30,48]. One class of these is even laid
out along columns of the dorsal periacqueductal gray (PAG [49]). Nevertheless,
any of these defensive manoeuvres would interrupt the ongoing chain of actions,
and this is what we modelled. Risk assessment and exploration are of most
obvious use in the face of uncertainty and ignorance, whereas conditioned
suppression, and thus the sort of inhibition that we consider, remains even
after substantial learning. It would certainly be worth going one stage further,
modelling the interruption in terms of a switch between different Markov
decision problems, with new information changing the transition and payoff
structures.
Tryptophan Depletion
One of our central results is the effect of an acute reduction in
α5HTafter learning with elevated α5HT has
taken place. In our model, this leads to a decrease in
behavioral inhibition of actions leading to negative states. Although specific
effects might arise from local manipulations of 5-HT concentrations or receptor
responsivity, key data come from the systemic manipulation associated with acute
TrD [50],
in which plasma levels of tryptophan and, at least in animals, central nervous
system levels of serotonin, are drastically reduced (by up to 90%).
Although the particular chains of thoughts analysed here have not been the
subject of experimental scrutiny, there is by now a considerable body of
literature on the effects of TrD on normal human functioning. In broad agreement
with our results, various effects have been related to decreased reward
processing [39,51,52], decreased
behavioral inhibition [44,53–57], rumination [21], facial fear recognition
[58],
and, more indirectly, increased aggressiveness [54,59,60].
Perhaps of most direct relevance to our implementation are the results of a
recent study which decoupled rewards from correct performance of an action from
the outcomes of the actions [61]. This study actually involved a
sophisticated assessment of the effects of TrD on reversal learning. However,
one way of viewing a portion of the results stems from an abstract
representation of the task. Subjects had to press one of two buttons (A or B) in
response to one of two stimuli (also called A and B), with presses associated
with A leading to a symbolic reward and presses associated with B leading to a
symbolic punishment. Critically, these outcomes were independent of the
rectitude of the subjects' responses, so they couldn't avoid the punishment by
making errors. In this case, subjects more often failed to press button B
correctly than button A, and this difference disappeared after TrD. This is
directly consistent with the present interpretation of serotoninergic inhibition
of actions that lead to aversive outcomes.
Famously, TrD does not have a uniform effect on all subjects. There is an
important genetic polymorphism in the 5-HT reuptake mechanism, with subjects
having the less efficient version generally showing greater effects
[52,57,62–66]. For this to be consistent with
our formulation, the difference in functional 5-HT levels before and after TrD
has to be greater in the subjects with less efficient reuptake. This in turn
might most simply be due to increased levels of 5-HT (and behavioral inhibition)
throughout development in carriers of the short 5HTTLPR allele. Perhaps related
to this is the finding that TrD produces a dose-dependent relapse of depressive
symptomatology in formerly depressed patients [18–20,41], or in patients with risk
factors such as a family history of depression [63] (although the three-way
interaction between TrD, 5HTTLPR, and past depression is hard to fit into this
framework [67]).
There is a significant body of work on the effects of serotonergic manipulations
on affective processing, particularly on processing of facial expressions
[58,68–70]. It is difficult
to interpret this work in our context for several reasons: first, there have
often been effects on recognition of specific aversive facial expressions (e.g.,
fear) but not others (e.g., disgust). Our model does not speak to these
distinctions. Second, in these tasks, subjects identify stimuli by pressing a
button. Thus, there is a Pavlovian association between certain buttons and the
aversive stimuli, and, interpreting these tasks in the same framework as we
interpreted the work of Cools et al. [61], one might predict that TrD
would increase rather than decrease accuracy. The precise effect, however, would
depend on the relative strength of the instructed and the reflexive Pavlovian
response, and on the antagonism between the responses. Indeed, both aspects have
been found: acute manipulation of serotonin increased recognition accuracy of
fearful faces with increasing serotonin [58,68,70], whereas a more chronic increase
in 5-HT (via SSRIs) yields a decrease in recollection of negative memories
[69].
Furthermore, while the exact relationship between behavioral inhibition and
amygdala activation still needs clarification, it is additionally possible that
increased amygdala activation may relate to perceptual mood congruency effects
[38]:
after disinhibition, thoughts often visit negative states, and it is possible
that this may affect prior expectations about stimulus which in turn could speed
up processing of negatively valenced information.
TrD (or indeed SSRIs) have not previously been used in tasks like the Markov
decision problem of the type we discussed. A direct prediction of the model is
that subjects trained under TrD would explore states less when tested in a
normal regime, while those trained under SSRIs would do so more (assuming that
SSRIs indeed elevate 5-HT levels). Similar predictions hold for subjects with
short or long alleles of the 5-HT reuptake mechanism on these tasks. This would
essentially represent a generalisation of the findings by Cools et al.
[61]
to the domain of sequential decisions. The tasks could use external, observable
actions; more directly, it would also be useful to monitor the execution of
affective trains of thought, and study the perturbation of this under
serotonergic manipulations. In designing such studies, it is important to bear
in mind the potentially opponent instrumental and Pavlovian effects, in just the
same way that boosting dopamine and monitoring the effects on negative
automaintenance may be confusing. Note that although there are various important
datasets as to the effects of TrD on simple probabilistic and delay-discounting
tasks [51,52,56,71–75], these studies do not encompass
the sorts of behavioral chains that we propose 5-HT to be able to halt.
Dopamine and Serotonin
One of the backdrops for the present theory was the extensive modeling of phasic
dopamine as a prediction error for future reward, and the results that (1) the
baseline firing rates of dopamine cells are insufficient to report prediction
errors for negative rewards (i.e., punishments); (2) the ample psychological
evidence for the existence of a pair of systems, one associated with appetitive
outcomes and the other with aversive outcomes; and (3) the evidence that at
least some aspects of 5-HT and dopamine are in mutual opposition. Indeed, based
on these data and the theories of Deakin and Graeff [6,7], Daw et al. [10] suggested that
serotonin rather than dopamine reports negative prediction errors based on an
antagonism between serotonin and dopamine at both a behavioral and
pharmacological level. For example, in rodents, 5-HT antagonises the general
excitatory effects of dopamine [4], the self-administration of
amphetamine and intracranial self-stimulation [76,77], the effects of dopamine on
appetitive learning [8], and the potentiation of appetitive learning by amphetamine
[78].
However, pure opponency is far too simple. For instance, there is by now
extensive evidence that 5-HT modulates dopaminergic activity both through
receptors in the ventral tegmental area and by modulation of distal release
sites, and that this modulation can occur in both inhibitory and excitatory
directions [4,77,79–89]. Even the rise
in 5-HT due to SSRIs has overall pro-dopaminergic effects, both at behavioral
and physiological levels [81,90–92], and there is one report that DA antagonists reverse the
antidepressant effect of SSRIs [93]. Further, there is evidence that
DA itself is released in many aversive circumstances [94,95], and is involved in aversively
motivated behaviors like avoidance [96–98].
In our terms, apart from the aspects of the interaction of dopamine and 5-HT that
were explored in Figures 5B
and S2,
there are a couple of other effects. First, inhibition in our model has the
consequence of increasing the average expected reward. As such, tonic dopamine,
which has been suggested to report such a quantity [46,47,99–101], would be increased when 5-HT
is boosted, and potentially vice versa [88,92]. This would compete with the
more direct effect that 5-HT inhibits actions, and particularly inhibits actions
supported by dopaminergic predictions or rewards [8,78,102], and thus high levels of 5-HT
might also depress levels of tonic dopamine, more in line with accounts that
stress the opponent role of dopamine and serotonin [4,10,103].
The second complexity (Boureau, personal communication) is that active defense
(such as active avoidance) requires energizing, and indeed appears to be
controlled by the (presumably dopaminergically reported) appetitive outcome of
reaching a state of safety rather than the (presumably serotonergically
mediated) outcome of leaving a state of fear. That is, it appears that dopamine
reports the rewards reaped from avoiding or controlling aversive outcomes
[15,94,104].
We mentioned the mirror notion that the relationship between 5-HT and inhibition
arising through aversive predictions is parallel to the obverse relationship for
dopamine and engagement/approach through appetitive predictions [32]. In this case,
appetitively directed chains of thoughts would be favored. Indeed, Smith et al.
[105,106], in their work
on the conditioned avoidance model of schizophrenia, suggested something rather
like this. In their account, dopamine controls the extent of search through a
forward model, although they did not couple this to dopamine's involvement in
appetitive prediction.
In all, disentangling and elucidating these varied relations between dopamine and
serotonin is a pressing task.
Depression and Anxiety
It would be reasonable to argue that the present model is more relevant to
anxiety than depression. There is at best a somewhat fuzzy distinction between
the two in terms of risk factors [107] and pharmacology
[108], and they are extraordinarily co-morbid [109]. There is also
no complete definition of either disease in terms of the sort of reinforcement
mechanisms that we have been considering.
While depressive (but not anxious) symptoms can be reinduced by TrD in a
subpopulation of patients, TrD it is not the only such manipulation, and it is
not effective in all patients. Patients who are responsive to
seratonin–norepinephrine reuptake inhibitors (SNRIs) are more
sensitive to catecholamine depletion by α-methyl-tyrosine
[110,111] than TrD, and a
recent report with a DA antagonist successfully re-induced depressive symptoms
in formerly depressed people [93]. The latter authors suggest that
DA may be a “final common path” for depression, and may
relate more to the depressive state than serotonin, which in turn may be more
important in defining a trait [15,112,113]. In addition, only
50% of formerly depressed subjects do respond to TrD [41,114], and a pooled analysis of 71
formerly depressed subjects found that previous response to SSRIs had less
predictive power for TrD response than chronicity of the depressive disorder and
sex [114]. As mentioned, resolving the actual relative contribution
of serotoninergic inhibition will be tricky.
Conclusions
In sum, the findings in this study argue for an involvement of the serotonin
reuptake mechanism in mood disorders such as anxiety and depression in the
following manner: due to a decreased efficiency of the transporter, increased
behavioral inhibition results in acquisition of overly optimistic values. Such
value functions are adaptive, but only in conjunction with strong behavioral
inhibition. On the other hand, they do render the individual highly sensitive to
large decreases in average experienced rewards when serotonin function is
reduced. This might underlie a (controversially) larger sensitivity to TrD and
SSRIs of persons with the short 5HTTLPR allele (see [115]). Returning to
the sequential decision-making tasks suggested above, this study would predict
that the short 5HTTLPR allele would be associated with more reflexive avoidance
of states predictive of punishment, and it may be possible to assess this with
differential effects of TrD on carriers with the short and long 5HTTLPR allele.
A further, more involved conjecture, which returns to the fact that serotonin is
not the sole causative agent in depression, is that it is the effects of reduced
5-HT on affective experience that leads to the various symptoms of depression,
acting via the otherwise normative operation of the multiple systems involved in
behavioral control. For instance, we have argued that the consequences of 5-HT
reduction include unexpected punishments, large negative prediction errors, and
a drop in average reward. These changes in the statistics of reward demand
explanation, for example in terms of a shift in the characteristics of the
environment, and should cause normative behavioral responses.
In particular, the unsignalled aversion that comes independent of the subject's
actions can be seen as a form of uncontrollable punishment. Uncontrollability
lies at the heart of an important characterization of depression centred around
learned helplessness [104,116].
We concentrated on the effects of reduced 5-HT rather than on the reasons for
this reduction. The obvious option is that it is a pathological result from
processes operating at a purely cellular level. However, it could also arise as
a normative meta-adaptation to the statistics of experienced punishments and
rewards. Formalizing this fully would require a more general theory of
inhibition—what level of inhibition is optimal? Tools for the
characterisation of the trade-off between accurate knowledge about a state's
value and the cost incurred in learning about it are already in existence
[34,117,118] and might be applicable to
aspects of the present case.
Supporting Information
A Deep Environment
Similar state space to Figure
1, but with a more explicitly deep structure. State in
mainly lead to
,
or back to themselves. The last states in each of the two chains (here
and
)
always preferentially lead to the outcome state and .
(32 KB PDF)
Inhibition in a Deep Environment
The outcomes are approached by sequentially
walking through K = 4 levels. Only
states lead to outcomes.
(A,D) True values without inhibition are shown by the black line. It is
constant for each level and valence, or illustration, as all outcomes were
assigned the same positive value (+1 or −1). The reward
of the states is zero and shown by the dash-dotted line. The grey point
display the estimated values of the states under inhibition
α5HT = 20. There is a positive bias in
all states, but it is more pronounced in the states with true negative
values. In (D), the dash-dotted line indicates that states
now carry reward −0.4, while states
carry reward +0.4. States
for k = {1,2,3} now have true
negative values, and
for k = {1,2,3} have true
positive values.
(B,E) Probabilities of ending thought sequence in or .
(C,F) Effect of preferentially choosing actions according to their valence on
the average value of states. The arrow indicates increasing
γ. In (C), larger
γ are advantageous, in (F), smaller
γ are better.
(210 KB PDF)
Impulsivity in a Deep Environment
(34 KB PDF)
We are grateful to Y-Lan Boureau, Roshan Cools, Nathaniel Daw, Hanneke Den Ouden,
Karl Friston, Michael Moutoussis, Jon Roiser, Barbara Sahakian, Douglas Steele,
Jonathan Williams, and Paul Willner for helpful discussions. We would also like to
thank anonymous reviewers for helpful comments.
AbbreviationsBIS
behavioral inhibition system
SSRI
selective serotonin reuptake inhibitor
TrD
tryptophan depletion
ReferencesSolomonRLCorbitJD1974An opponent-process theory of motivation. i. temporal dynamics of
affect.81119145DickinsonADearingMF1979Appetitive-aversive interactions and inhibitory processes.In:DickinsonABoakesRAHillsdale (New Jersey)Erlbaum203231DickinsonABalleineB2002The role of learning in the operation of motivational systems.In:GallistelRNew YorkWiley497534CarterCJPycockCJ1978Differential effects of central serotonin manipulation on
hyperactive and stereotyped behaviour.23953960CostallBHuiSCNaylorRJ1979The importance of serotonergic mechanisms for the induction of
hyperactivity by amphetamine and its antagonism by intra-accumbens
(3,4-dihydroxy-phenylamino)-2-imidazoline (dpi).18605609DeakinJFW1983Roles of brain serotonergic neurons in escape, avoidance and
other behaviors.4356377DeakinJFWGraeffFG19915-HT and mechanisms of defence.5305316FletcherPJ1996Injection of 5-HT into the nucleus accumbens reduces the effects
of d-amphetamine on responding for conditioned reward.1266269KapurSRemingtonG1996Serotonin–dopamine interaction and its relevance to
schizophrenia.153466476DawNDKakadeSDayanP2002Opponent interactions between serotonin and dopamine.15603616EspositoE2006Serotonin–dopamine interaction as a focus of novel
antidepressant drugs.7177185SoubriéP1986Reconciling the role of central serotonin neurons in human and
animal behaviour.9319364GrayJA1991Cambridge (United Kingdom)Cambridge University Press432SchmajukNAGrayJALamYW1996Latent inhibition: A neural network approach.22321349WillnerP1985New YorkJohn Wiley and Sons597GraeffFGGuimaraesFSDe AndradeTGCSDeakinJFW1998Role of 5HT in stress, anxiety and depression.54129141MaierSFWatkinsLR2005Stressor controllability and learned helplessness: The roles of
the dorsal raphe nucleus, serotonin, and corticotropin-releasing factor.29829841YoungSNSmithSEPihlROErvinFR1985Tryptophan depletion causes a rapid lowering of mood in normal
males.87173177DelgadoPLCharneyDSPriceLHAghajanianGKLandisH1990Serotonin function and the mechanism of antidepressant action:
Reversal of antidepressant-induced remission by rapid depletion of plasma
tryptophan.47411418MorenoFAGelenbergAJHeningerGRPotterRLMcKnightKM1999Tryptophan depletion and depressive vulnerability.46498505SmithKAFairburnCGCowenPJ1999Symptomatic relapse in bulimia nervosa following acute tryptophan
depletion.56171176CaspiASugdenKMoffittTETaylorACraigIW2003Influence of life stress on depression: Moderation by a
polymorphism in the 5-HTt gene.301386389MillanMJ2006Multi-target strategies for the improved treatment of depressive
states: Conceptual foundations and neuronal substrates, drug discovery and
therapeutic application.110135370SibilleELewisDA2006Sert-ainly involved in depression—but when?.163811EstesWSkinnerB1941Some quantitative aspects of anxiety.29390400FletcherPJ1995Effects of combined or separate 5,7-dihydroxytryptamine lesions
of the dorsal and median raphe nuclei on respondingmaintained by a DRL 20s
schedule of food reinforcement.6754554FletcherPJKorthKM1999Activation of 5-HT1B receptors in the nucleus accumbens reduces
amphetamine-induced enhancement of responding for conditioned reward.142165174GraeffFG2002On serotonin and experimental anxiety.163467476GrayJAMcNaughtonN2000OxfordOxford University Press424McNaughtonNCorrPJ2004A two-dimensional neuropsychology of defense: Fear/anxiety and
defensive distance.28285305BrelandKBrelandM1961The misbehavior of organisms.16681684DayanPNivYSeymourBDawND2006The misbehavior of value and the discipline of the will.1911531160KnuthDMooreR1975An analysis of alpha-beta pruning.6293326BaumEBSmithWD1997A Bayesian approach to relevance in game playing.97195242SuttonRSBartoAG1998Cambridge (Massachusetts)MIT Press322WatkinsCJCH1989Cambridge (United Kingdom)King's College, Cambridge UniversityAvailable at: http://www.cs.rhbnc.ac.uk/home/chrisw/new_thesis.pdf.
Accessed 6 December 2007.RachmanSArntzA1991The overprediction and underprediction of pain.11339355BlaneyPH1986Affect and memory: A review.99229246KlaassenTRiedelWJDeutzNEVan PraagHM2002Mood congruent memory bias induced by tryptophan depletion.32167172Nolen-HoeksemaS1991Responses to depression and their effects on the duration of
depressive episodes.100569582SmithKAFairburnCGCowenPJ1997Relapse of depression after rapid depletion of tryptophan.249915919DoyaK2000Metalearning, neuromodulation and emotion.In:HatanoGOkadaNTaHAmsterdamElsevier Science101104TanakaSCDoyaKOkadaGUedaKOkamotoY2004Prediction of immediate and future rewards differentially
recruits cortico-basal ganglia loops.7887893SchweighoferNShishidaKHanCEOkamotoYTanakaSC2006Humans can adopt optimal discounting strategy under real-time
constraints.2e152SchweighoferNTanakaSCDoyaK2007Serotonin and the evaluation of future rewards: Theory,
experiments, and possible neural mechanisms.1104289300NivYDawNDayanP2005How fast to work: Response vigor, motivation and tonic dopamine.In:Cambridge (Massachusetts)MIT Press10191026NivYDawNDJoelDDayanP2007Tonic dopamine: Opportunity costs and the control of response
vigor.191507520BlanchardDCBlanchardRJ1988Ethoexperimental approaches to the biology of emotion.394368BandlerRShipleyMT1994Columnar organization in the midbrain periaqueductal gray:
Modules for emotional expression?.17379389BellCAbramsJNuttD2001Tryptophan depletion and its implications for psychiatry.178399405MurphyFCSmithKACowenPJRobbinsTWSahakianBJ2002The effects of tryptophan depletion on cognitive and affective
processing in healthy volunteers.1634253RoiserJPBlackwellADCoolsRClarkLRubinszteinDC2006Serotonin transporter polymorphism mediates vulnerability to loss
of incentive motivation following acute tryptophan depletion.3122642272LeMarquandDGBenkelfatCPihlROPalmourRMYoungSN1999Behavioral disinhibition induced by tryptophan depletion in
nonalcoholic young men with multigenerational family histories of paternal
alcoholism.15617711779BjorkJMDoughertyDMMoellerFGSwannAC2000Differential behavioral effects of plasma tryptophan depletion
and loading in aggressive and nonaggressive men.22357369DeakinJF2003Depression and antisocial personality disorder: Two contrasting
disorders of 5HT function.647993AndersonIMRichellRABradshawCM2003The effect of acute tryptophan depletion on probabilistic choice.1737HaywardGGoodwinGMCowenPJHarmerCJ2005Low-dose tryptophan depletion in recovered depressed patients
induces changes in cognitive processing without depressive symptoms.57517524HarmerCJRogersRDTunbridgeECowenPJGoodwinGM2003Tryptophan depletion decreases the recognition of fear in female
volunteers.167411417WalshMTDinanTG2001Selective serotonin reuptake inhibitors and violence: A review of
the available evidence.1048491LeschKPMerschdorfU2000Impulsivity, aggression, and serotonin: A molecular
psychobiological perspective.18581604CoolsRRobinsonOJSahakianB2007Acute tryptophan depletion in healthy volunteers enhances
punishment prediction but does not affect reward prediction.In press.LeschKPBengelDHeilsASabolSZGreenbergBD1996Association of anxiety-related traits with a polymorphism in the
serotonin transporter gene regulatory region.27415271531NeumeisterAKonstantinidisAStastnyJSchwarzMJVitouchO2002Association between serotonin transporter gene promoter
polymorphism (5HTTLPR) and behavioral responses to tryptophan depletion in
healthy women with and without family history of depression.59613620RoiserJPRogersRDCookLJSahakianBJ2006The effect of polymorphism at the serotonin transporter gene on
decision-making, memory and executive function in ecstasy users and
controls.188213227HaririARHolmesA2006Genetics of emotional regulation: The role of the serotonin
transporter in neural function.10182191RoiserJPMüllerUClarkLSahakianBJ2007The effects of acute tryptophan depletion and serotonin
transporter polymorphism on emotional processing in memory and attention.10449461NeumeisterAHuXZLuckenbaughDASchwarzMNugentAC2006Differential effects of 5-HTTLPR genotypes on the behavioral and
neural responses to tryptophan depletion in patients with major depression
and controls.63978986HarmerCJBhagwagarZPerrettDIVöllmBACowenPJ2003Acute SSRI administration affects the processing of social cues
in healthy volunteers.28148152HarmerCJShelleyNCCowenPJGoodwinGM2004Increased positive versus negative affective perception and
memory in healthy volunteers following selective serotonin and
norepinephrine reuptake inhibition.16112561263BrowningMReidCCowenPJGoodwinGMHarmerCJ2007A single dose of citalopram increases fear recognition in healthy
subjects.21684690RogersRDBlackshawAJMiddletonHCMatthewsKHawtinK1999Tryptophan depletion impairs stimulus-reward learning while
methylphenidate disrupts attentional control in healthy young adults:
Implication for the monoaminergic basis of impulsive behaviour.146428491MobiniSChiangTJAl-RuwaiteaASHoMYBradshawCM2000Effect of central 5-hydroxytryptamine depletion on inter-temporal
choice: A quantitative analysis.149313318MobiniSChiangTJHoMYBradshawCMSzabadiE2000Effects of central 5-hydroxytryptamine depletion on sensitivity
to delayed and probabilistic reinforcement.152390397RogersRDTunbridgeEMBhagwagarZDrevetsWCSahakianBJ2003Tryptophan depletion alters the decision-making of healthy
volunteers through altered processing of reward cues.28153162CoolsRBlackwellAClarkLMenziesLCoxS2005Tryptophan depletion disrupts the motivational guidance of
goal-directed behavior as a function of trait impulsivity.301362RedgraveP1978Modulation of intracranial self-stimulation behaviour by local
perfusions of dopamine, noradrenaline and serotonin within the caudate
nucleusand nucleus accumbens.155277295HigginsGAFletcherPJ2003Serotonin and drug reward: Focus on 5-HT2c receptors.480151162FletcherPJKorthKMChambersJW1999Selective destruction of brain serotonin neurons by
5,7-dihydroxytryptamine increases responding for a conditioned reward.147291299ParsonsLHJusticeJBJr1993Perfusate serotonin increases extracellular dopamine in the
nucleus accumbens as measured by in vivo microdialysis.606195199GallowayMPSuchowskiCSKeeganMJHjorthS1993Local infusion of the selective 5ht-1b agonist cp-93,129
facilitates striatal dopamine release in vivo.159092D'AquilaPSColluMGessaGLSerraG2000The role of dopamine in the mechanism of action of antidepressant
drugs.405365373Di MatteoVCacchioMDi GiulioCEspositoE2002Role of serotonin(2C) receptors in the control of brain
dopaminergic function.71727734Di MatteoVDi GiovanniGDi MascioMEspositoE1999SB 242084, a selective serotonin2C receptor antagonist, increases
dopaminergic transmission in the mesolimbic system.3811951205GobertARivetJLejeuneFNewman-TancrediAAdhumeau-AuclairA2000Serotonin (2C) receptors tonically suppress the activity of
mesocortical dopaminergic and adrenergic, but not serotonergic, pathways: A
combined dialysis and electrophysiological analysis in the rat.36205221GrottickAFletcherPHigginsG2000Studies to investigate the role of 5-HT2C receptors on
cocaine-and food-maintained behavior.29511831191BortolozziADaz-MataixLScorzaMCeladaPArtigasF2005The activation of 5-HT 2A receptors in prefrontal cortex enhances
dopaminergic activity.9515971607Diaz-MataixLScorzaMBortolozziATothMCeladaP2005Involvement of 5-HT1A receptors in prefrontal cortex in the
modulation of dopaminergic activity: Role in atypical antipsychotic action.251083110843BenaliouadFKapurSRompréPP2007Blockade of 5-HT2a receptors reduces haloperidol-induced
attenuation of reward.32551561FletcherPSinyardJHigginsG2006The effects of the 5-HT 2C receptor antagonist SB242084 on
locomotor activity induced by selective, or mixed, indirect serotonergic and
dopaminergic agonists.187515525SerraGArgiolasAKlimekVFaddaFGessaGL1979Chronic treatment with antidepressants prevents the inhibitory
effect of small doses of apomorphine on dopamine synthesis and motor
activity.25415423BessonAPrivatAMEschalierAFialipJ1999Dopaminergic and opioidergic mediations of tricyclic
antidepressants in the learned helplessness paradigm.64541548Sasaki-AdamsDMKelleyAE2001Serotonin–dopamine interactions in the control of
conditioned reinforcement and motor behaviour.25440452WillnerPHaleASArgyropoulosS2005Dopaminergic mechanism of antidepressant action in depressed
patients.863745CabibSPuglisi-AllegraS1996Stress, depression and the mesolimbic dopamine system.128331342HorvitzJC2000Mesolimbocortical and nigrostriatal dopamine responses to salient
non-reward events.96651656BeningerRMasonSPhillipsAFibigerH1980The use of extinction to investigate the nature of
neuroleptic-induced avoidance deficits.691118PrinssenEPMKlevenMSKoekW1996Effects of dopamine antagonists in a two-way active avoidance
procedure in rats: Interactions with 8-OH-DPAT, ritanserin, and prazosin.128191197SmithALiMBeckerSKapurS2006Dopamine, prediction error and associative learning: a
model-based account.176184NivYDawNDDayanP2006Choice values.9987988FlorescoSBWestARAshBMooreHGraceAA2003Afferent modulation of dopamine neuron firing differentially
regulates tonic and phasic dopamine transmission.6968973GotoYGraceAA2005Dopaminergic modulation of limbic and cortical drive of nucleus
accumbens in goal-directed behavior.8805812FletcherPJAzampanahAKorthKM2002Activation of 5-HT(1B) receptors in the nucleus accumbens reduces
self-administration of amphetamine on a progressive ratio schedule.71717721UnglessMA2004Dopamine: The salient issue.27702706HuysQJM2007Reinforcers and control.Gatsby Computational Neuroscience Unit, UCLUniversity of London (United Kingdom)Available at: http://www.gatsby.ucl.ac.uk/~qhuys/pub.html. Accessed 6
December 2007.SmithALiMBeckerSKapurS2004A model of antipsychotic action in conditioned avoidance: A
computational approach.2910401049SmithAJBeckerSKapurS2005A computational model of the functional role of the
ventral-striatal d2 receptor in the expression of previously acquired
behaviors.17361395HettemaJMNealeMCMyersJMPrescottCAKendlerKS2006A population-based twin study of the relationship between
neuroticism and internalizing disorders.163857864ResslerKJNemeroffCB2000Role of serotonergic and noradrenergic systems in the
pathophysiology of depression and anxiety disorders.12219KaufmanJCharneyD2000Comorbidity of mood and anxiety disorders.126976DelgadoPL2000Depression: The case for a monoamine deficiency.61Supplement 6711IversenL2005The monoamine hypothesis of depression.In:LicinioJWongMLWeinheim (Germany)Wiley7186HeinzA1999Anhedonie—nosologieübergreifendes Korrelat
einer Dysfunktion des dopaminergen Verstärkungssystem?.70391398WillnerP2002Dopamine and depression.In:ChiaraGDBerlinSpringer387416BooijLder DoesWVBenkelfatCBremnerJDCowenPJ2002Predictors of mood response to acute tryptophan depletion: A
reanalysis.27852861KraftJPetersESlagerSJenkinsGReinaldaM2007Analysis of association between the serotonin transporter and
antidepressant response in a large clinical sample.61734742AbramsonLYAlloyLBHoganMEWhitehouseWGCornetteM1998Suicidality and cognitive vulnerability to depression among
college students: A prospective study.21473487DeardenRFriedmanNRussellS1998Bayesian Q-learning.In:StockholmAAAI Press761768DeardenRFriedmanNAndreD1999Model-based Bayesian exploration.In:StockholmAAAI Press150159