Skip to main content
Advertisement
  • Loading metrics

Individualised aspiration dynamics: Calculation by proofs

  • Bin Wu ,

    Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

    bin.wu@bupt.edu.cn

    Affiliation School of Sciences, Beijing University of Posts and Telecommunications, Beijing, China

  • Lei Zhou

    Roles Formal analysis, Funding acquisition, Investigation, Resources, Software, Validation, Writing – review & editing

    Affiliations Center for Systems and Control, College of Engineering, Peking University, Beijing, China, Department of Ecology and Evolutionary Biology, Princeton University, Princeton, New Jersey, United States of America

Abstract

Cooperation is key for the evolution of biological systems ranging from bacteria communities to human societies. Evolutionary processes can dramatically alter the cooperation level. Evolutionary processes are typically of two classes: comparison based and self-evaluation based. The fate of cooperation is extremely sensitive to the details of comparison based processes. For self-evaluation processes, however, it is still unclear whether the sensitivity remains. We concentrate on a class of self-evaluation processes based on aspiration, where all the individuals adjust behaviors based on their own aspirations. We prove that the evolutionary outcome with heterogeneous aspirations is the same as that of the homogeneous one for regular networks under weak selection limit. Simulation results further suggest that it is also valid for general networks across various distributions of personalised aspirations. Our result clearly indicates that self-evaluation processes are robust in contrast with comparison based rules. In addition, our result greatly simplifies the calculation of the aspiration dynamics, which is computationally expensive.

Author summary

Cooperation is the cornerstone to understand how biological systems evolve. Previous studies have shown that cooperation is sensitive to the details of evolutionary processes, even if all the individuals update strategies in the same way. Here we propose a class of updating rules driven by self-evaluation, where each individual has its personal aspiration. The evolutionary outcome is the same as if all the individuals adopt the same aspiration for regular networks, provided the selection intensity is weak enough. In addition, we provide a simple numerical method to identify the favored strategy. Our result shows a very robust class of strategy updating rules. And it implies that complexity in updating rules does not necessarily lead to the sensitivity of evolutionary outcomes.

Introduction

Cooperation is forgoing the focal individual’s own interest to benefit others. It is ubiquitous across every level of biological systems. Yet cooperation seemingly contradicts the evolution theory, since free riders, who contribute nothing, are better off than cooperators in fitness. The gap has attracted considerable attention to tackle how and why cooperation evolves [1].

A pairwise game, in which there are only two players, is widely employed to model the conflicts between cooperation and defection. The well-known Prisoners’ Dilemma is a pairwise game [2]. While the dyadic games capture the conflicts between cooperators and defectors, they neglect that the interactions in the real world usually involve more than two individuals. For example, all the nations are involved to reduce the emission of greenhouse gases, leading to a multiplayer collective risk dilemma [35]. More than two yeast cells interact to form a multicellularity when the resource is rare, resulting in a multiplayer game [6]. Thus, it is natural to adopt multiplayer games as a paradigm to study cooperation.

The last decade has seen an intensive study on the mechanisms promoting cooperation including group selection, direct reciprocity and spatial reciprocity [714]. In particular, spatial reciprocity attracts considerable attention with the development of network science. Spatial structure is typically described as networks, where nodes represent individuals and links social ties. Individuals on networks play games only with their neighbours. In this case, cooperators form clusters not only to invade but also to resist a population of defectors [13]. Consequently, the population structure promotes both the emergence and the stabilization of cooperation [15]. Yet the evolutionary outcome can be dramatically altered on networks, even if some detail of the evolutionary process changes: in degree-homogeneous networks, Death-birth process can promote cooperation for the Prisoners’ Dilemma whereas Birth-death process cannot [13]; In degree-heterogeneous networks, the fate of cooperation relies on the rate at which interactions occur and how interactions transform into the fitness of strategies [16]. In fact, the evolutionary outcome is not robust for different evolutionary processes even in the well-mixed population where all the individuals interact with everyone else [17, 18]. Yet, all of the above works assume that all the individuals adopt the same strategy updating rule. In reality, it is a personal trait to adjust one’s own behavior. In other words, individuals are equipped with their own updating rules, resulting in a great heterogeneity in strategy updating rule across the population. Considering that the details of evolutionary processes would greatly alter the evolutionary outcome, the fate of strategies is expected to be extremely sensitive to the heterogeneity in updating rules.

Here, we study aspiration dynamics of multiplayer games on regular networks and investigate how the heterogeneity in aspiration levels alters the evolutionary outcome. The idea of aspiration is self-evaluation: an individual is likely to keep its strategy if its aspiration is met by the payoff, and to switch otherwise. Aspiration dynamics are widely present in behavioural ecology. For example, experienced ants go back to their nest based on their own chemical trails [19] or by counting their own previous steps [20]. In our manuscript, every individual has its personal aspiration. We show that cooperation is more abundant over defection for all possible personal aspiration levels provided cooperation is advantageous for a homogeneous aspiration level. Furthermore, we analytically figure out the condition under which cooperation prevails, which is also true for all two-strategy multiplayer games. On the one hand, our result clearly shows that heterogeneity in aspiration does not alter the fate of strategies, in contrast with previous results that evolutionary outcome is sensitive to the details of the evolutionary processes [18, 21]. On the other hand, the result also provides computational savings when identifying the conditions under which one strategy is more abundant than the other.

Models

Let us consider a two-strategy d-player game. An individual using strategy A obtains payoff ak when encountering k coplayers using strategy A. And an individual using strategy B gets payoff bk when facing k coplayers using strategy A. The number of coplayers adopting strategy A, i.e., k, ranges from 0 to d − 1. The game is given by the following payoff table [22]:

In a well-mixed population with finite size N, an individual randomly chooses d − 1 individuals from the entire population to play with based on Table 1. It results in a hypergeometric distribution of d − 1 players with respect to the number of individuals using strategy A. In other words, individuals interact globally in the well-mixed population. The interaction between individuals, however, may be limited by territorial constraints. Thus we concentrate on the more realistic local interaction. We assume that all the individuals are located on a regular network of size N with degree d − 1. In this case, each of the N individuals in the population has d − 1 neighbours. At each time, an individual, namely i ranging from 1 to N, is randomly selected. It plays the d-player game with its d − 1 neighbours, and obtains payoff πi. The focal individual i is likely to switch its strategy if its current payoff πi does not meet its aspiration ei.

We assume that every individual has its own way of switching. To this end, we allow that individual i, ranging from 1 to N, switches its strategy with a probability given by its own decision making function gi. Here gi is a function of β(eiπi), with selection intensity β ≥ 0 [23]. All through our manuscript, we assume that the decision making function gi is

  1. differentiable on the real line with its derivative strictly positive, i.e., ;
  2. between zero and one, i.e., 0 < gi < 1.

For 1), the positive derivative of the decision making function implies that the decision making function is strictly increasing. In this case, an individual whose aspiration is not met is more likely to change its strategies. For 2), the bound between zero and one yields that gi is a probability and that there is always a non-zero probability of changing strategies.

The aspiration dynamics on a network is a Markov chain. The state space of the Markov chain is S = {(s1, s2…, sN)|si ∈ {A, B}}, where si is 1 if the strategy of individual i is A, and is 0 otherwise. is the number of individuals using strategy A. There is no absorbing state of the Markov chain, since the decision making functions we propose ensure that all the individuals could alter their strategies at any time. The Markov chain is ergodic, thus there is a stationary distribution κs, sS. The average abundance of strategy A in the stationary regime is obtained by . Here κs is a function of selection intensity. We are interested in the condition under which strategy A is favored. In other words, we try to figure out the condition under which the average abundance of strategy A is more abundant than that of strategy B. i.e., .

(Table 2) summarizes key notations in this section. It is noteworthy that both the aspiration and the decision making function are personal traits, resulting in the heterogeneity in decision making.

An equivalent condition that strategy A is favored

We show that the average abundance of strategy A is one half when the selection intensity vanishes. And this remains true for all possible individualised aspirations ei and decision making functions provided gi(0) > 0 for all i = 1, 2…, N (see Section The average abundance is 1/2 for vanishing selection intensity in S1 File. for the proof). In particular, it is true for our decision making functions, since our decision making function gi is positive for the real line. Intuitively, no individual has any bias towards any strategy for the vanishing selection intensity, although they update strategies in their own ways. Thus strategy A is more abundant than strategy B if and only if the abundance of strategy A is more than that of its neutral case, which holds for all possible individualised aspirations, aspiration-based decision making functions and selection intensities.

The criterion is a linear inequality

In the following, we introduce two lemmas, which are crucial to the main result.

Lemma 1 For the decision making function gi with 0 < gi < 1 and positive derivative on the real line (i = 1, 2,…, N), under weak selection limit, there exist parameters αk, ωk (k = 0, 1, …, d − 1) and ϕi (i = 1, 2, …, N), which are neither dependent on the payoff entries nor the aspiration levels, such that if (1) strategy A is more abundant than strategy B.

We outline the proof: The differentiability of the decision making function ensures that the average abundance of strategy A, i.e., , is differentiable with respect to the selection intensity [24]. Thus the average abundance of strategy A under weak selection limit is given by (2) The first term is the average abundance of strategy A when the selection intensity vanishes, it is one half based on Section The average abundance is 1/2 for vanishing selection intensity in S1 File. Therefore, the average abundance of strategy A is more than that in the neutral case if is positive. Thus is the condition under which strategy A is more abundant than strategy B. We show that is linear in payoff and aspiration level with no constant term, i.e., a term does not contain payoff entries or aspirations (Section The criterion is a linear inequality of payoffs and aspirations in S1 File).

Lemma 2 If all the payoff entries are the same, i.e., there exists a constant h such that ak = bk = h for all k = 0, 1…, d − 1, then the average abundance of strategy A is one half for all possible individualised aspirations and all aspiration-based decision making functions fulfilling gi(0) > 0, i = 1, 2, …, N and all selection intensities.

(See Section The average abundance is 1/2 for neutral mutants for any selection intensity in S1 File for the proof.)

Lemma 2 suggests that if a neutral mutant arises, the heterogeneity in the decision updating rule and aspiration level does not alter the evolutionary outcome, compared with the homogeneous case. In particular, Lemma 2 is true for any selection intensity. Intuitively, both strategies are identical in payoffs when all the payoff entries are the same. It resembles a neutral mutant in population genetics [25]. Furthermore, each individual shows no bias towards any of the two strategies, although everyone has its own way to update strategies. By symmetry, the two strategies should be equally abundant.

Results

The condition is independent of individualised aspirations

Theorem 1 Consider a regular network with degree d − 1, and the decision making function with 0 < gi < 1 and positive derivative on the real line (i = 1, 2, …, N), in the limit of weak selection, there exist d parameters σks (k = 0, 1…, d − 1) which are neither dependent on payoff entries nor aspiration levels such that if (3) then strategy A is more abundant than strategy B.

Proof 1 For the payoff table with all the entries being zero, i.e., ak = bk = 0 for 0 ≤ kd − 1, strategy A and B are equally abundant for any selection intensity by Lemma 2. Thus the two strategies are equally abundant under weak selection limit. On the one hand, strategy A is not more abundant than strategy B. Thus there exist N parameters ϕis (i = 1, 2…, N) which are neither dependent on the aspiration levels nor payoff entries such that holds by Lemma 1. On the other hand, strategy B is not more abundant than strategy A. Thus holds with the same argument by Lemma 1. Therefore, it yields that (4) Note that the parameters ϕis, i = 1, 2, …, N, are independent of the aspirations levels. And Eq (4) is valid for any eis (i = 1, 2, …, N). In particular, for any j (1 ≤ jN), let ei = δij, where δij is the Kronecker Delta. We have that ϕj = 0 by Eq (4).

Therefore, based on Lemma 1, the condition under which strategy A is more abundant than strategy B is (5) Switching the name of A and B, by Lemma 1, strategy B is more abundant than strategy A if (6) holds.

On the one hand, the linear combination of Inequality (6) is proportional to the first-order derivative of the abundance of strategy B; On the other hand, the sum of the abundance of strategy A and B is always one for any selection intensity. Thus the first-order derivative of the sum of the two abundances is zero. In other words, the first-order derivative of the abundance of strategy A is always different from that of strategy B in sign, provided neither of them is zero. Thus, strategy A is more abundant than strategy B if and only if Inequality (6) does not hold.

Therefore, strategy A is more abundant than strategy B if (7) or (8) Both Inequality (5) and (8) are the condition under which strategy A is more abundant than strategy B. Thus there exists a positive rescaling factor λ > 0 such that they are mapped to each other, i.e., αk = −λωd−1−k and ωk = −λαd−1−k hold for 0 ≤ kd − 1. In particular, we have α0 = −λωd−1 and ωd−1 = −λα0. Thus λ2 = 1. We have that λ = 1 since the constraint λ > 0. This yields that αk = −ωd−1−k for k = 0, 1,…, d − 1. In this way, Inequality (5) becomes (9) By Lemma 1, αk is not dependent on the aspirations or the payoff entries. Therefore σk is neither dependent on the aspirations nor the payoff entries.

With the aid of Lemma (1) and Lemma (2), we obtain the condition under which strategy A is more abundant than strategy B. Herein, Lemma 1 states that the criterion under which a strategy is more abundant than the other is determined by both payoff entries and individualised aspirations. Yet Lemma 2 implies that the criterion is not dependent on the aspiration levels at all, and it is the cornerstone of the proof.

The theorem indicates that the criterion under which strategy A is favored is a simple linear inequality (3). The coefficients of the inequality are neither dependent on the payoff entries nor the aspirations, but are determined by the decision making functions, the way of payoff collecting, population size and the population structure.

The coefficients of the inequality do not depend on the payoff entries, which is similar to the σ-rule of the mutation-selection dynamics [24, 26]. The independence ensures that the coefficient σk is obtained by adopting the payoff table with ak = 1 and all the rest being zero. Thus it opens up an avenue to compute the evolutionary dynamics under weak selection limit [27].

The theorem ensures that the coefficients do not depend on the aspiration levels at all. Simulations also show that the abundance of strategy A almost keeps invariant for a diverse class of distributions of individual aspirations (Figs (1) and (2)). It is counter-intuitive that the coefficients are independent of the aspiration levels. In fact, it is well-known for games on networks that the heterogeneity of personal traits alters the evolutionary outcome [2832]. For example, cooperation can be greatly promoted if individuals have heterogeneous neighborhood sizes [33]. Aspirations are part of the individual’s updating rule. Thus the heterogeneity in updating rule should have changed the evolutionary outcome for some games. The intuitive understanding resorts to weak selection limit. On the one hand, the evolutionary outcome is not determined by the aspiration under weak selection limit when the aspiration is homogeneous in the population [34]. On the other hand, the driving force of the strategy updating is stochastic neutral drift under weak selection. Therefore, the effect of heterogeneity in aspiration is weakened, resulting in a similar dynamics with the case of homogeneous aspiration. In fact, individual i uses strategy B more often if the switching probability from A to B is greater than the other way around. In other words, strategy B is more abundant for individual i if g(β(eiπA)) > g(β(eiπB)), i.e., πB > πA under weak selection. This suggests that the criterion should be independent of the individualised aspiration, consistent with our theorem.

thumbnail
Fig 1. Average abundance of strategy A for a three-player game with individualised aspirations.

Personal aspirations are randomly assigned based on a uniform distributions on the interval [0, 1] (left panels) or [0, 5] (right panels). Within each class, two sets of aspirations are assigned to represent two populations with different aspirations from the same distribution. In this way, we established two kinds of heterogeneity in aspiration: one resorts to the underlying distribution, and the other is based on the actual values sampled from the same distribution. Intuitively, both of the two heterogeneities would alter the evolutionary outcome. Simulations in line with our theorem, however, show that both kinds of the heterogeneity in aspiration lead to the identical abundance in A for both the well-mixed population (upper panels) and that on rings (lower panels). We illustrate other details of the simulation in the following. In well-mixed populations, the focal individual randomly chooses two individuals from the rest of the population, whereas it plays only with the nearest two neighbours on the ring. In the beginning, we randomly set 5% of the population to be of strategy A and the rest to be of strategy B. For each data point, it is the mean of 20 independent runs. In each run, we iterate the evolutionary process for 1 × 108 generations. The average abundance of strategy A is obtained by averaging abundance of strategy A over the last 5 × 107 generations. The population size N = 100. And the payoff entries are a0 = 3, a1 = 2, a2 = 1, b0 = 4, b1 = 1 and b2 = 1, respectively.

https://doi.org/10.1371/journal.pcbi.1006035.g001

thumbnail
Fig 2. Average abundance of strategy A as a function of the payoff entry a0.

Our simulation results clearly show that strategy A is more abundant than strategy B if a0 is approximately greater than 2.0 for regular networks (k = 2) (Panel (a)). This is in perfect agreement with our calculation based on the theorem, i.e., inequality a0 + 2a1 + a2 > b0 + 2b1 + b2 with a1 = 2, a2 = 1, b0 = 4, b1 = 1 and b2 = 1. Note that the criterion a0 > 2 is valid for aspiration across various distributions. It implies that the criterion to favor one strategy over the other is independent of the individualised aspiration, as stated in the theorem. Furthermore, the criterion also holds beyond regular networks (Panel (a)), namely, on random (Panel (b)) and scale-free networks (Panel (c)). It suggests that the criterion can be extrapolated to general population structures. The details of the simulations are as follows: The minimum degree of all the networks is set to be two such that all the individuals have enough neighbours to play the three-player game with. Personal aspirations are randomly assigned in a population with homogeneous aspiration ei = 2 for all i = 1, 2…N (blue ◯), and a population with heterogeneous aspirations generated based on uniform distribution on the interval [0, 5] (red □), bimodal distribution with (orange ◊), and power-law distribution with probability density function f(x) = 2x−3 (purple △). Here, stands for the normal distribution with mean 2.5 and standard deviation 0.5. In addition, the minimum value of aspiration sampled from the power-law distribution is 1.0. In the beginning, we randomly set 45% of the population to be of strategy A and the rest to be of strategy B. At each time step, the focal individual randomly chooses two individuals from its neighborhood and play a single three-player game with them to obtain the payoff. Fermi function is employed as the decision making function for all the individuals. Each data point is the mean of the average abundance of strategy A calculated from three independent runs (5 × 109 samples in each run, 1.5 × 1010 samples in total). In each run, we start sampling after a relaxation time of 5 × 107 time steps. The average abundance of strategy A is obtained by averaging the abundance of strategy A over 5 × 109 time steps. The population size N = 1000. The selection intensity β = 0.005.

https://doi.org/10.1371/journal.pcbi.1006035.g002

The coefficients are only dependent on individuals’ decision making functions, population size, the way of payoff collecting and the underlying population structure. In particular, they are determined by all the decision making functions gi, i = 1, 2⋯, N. Thus personalised updating rules might alter the evolutionary outcome, which is not the focus of our manuscript.

To sum up, the theorem shows that personalised updating rules might alter the evolutionary outcome, but not via heterogeneous aspiration levels.

Computation of the criterion for one strategy to be favored over the other

So far, we have obtained the condition under which a strategy is more abundant than the other, when everyone is equipped with its own aspiration. The condition is a linear inequality with as many coefficients as the game size. Further investigations show that those coefficients are always non-negative, and at least one of them is positive (see Section Non-negativity of the coefficients in S1 File). It mirrors the non-negativity of the structure coefficients of the σ-rule for mutation-selection dynamics [35].

In the following, we estimate the d coefficients, i.e., σk k = 0, 1, 2…, d − 1. For simplicity, we assume that all the individuals adopt the Fermi function as the decision making function, i.e. gi(x) = [1 + exp(−x)]−1 for i = 1, 2⋅⋅⋅N.

By our theorem, the coefficients for a population with individualised aspirations are exactly the same as the coefficients for a population with the homogeneous aspiration. Therefore, the coefficients for the population with heterogeneous aspirations can be calculated via those with a homogeneous aspiration (Section Calculation of coefficients in S1 File). And we obtain that (10) Eq (10) implies that the condition under which strategy A is more abundant on regular networks is exactly the same as that when the population is well-mixed. Generally, Eq (10) holds, provided all the individuals have the same decision making function with non-vanishing derivative at zero. (see Section Estimating coefficients S1 File).

Numerically, we estimate the coefficients for a three-player game on both well-mixed populations and rings. In well-mixed populations, the focal individual gets its payoff by playing games with two coplayers randomly selected from the rest of the population while on rings, its payoff is derived from interacting with its two nearest neighbours. We set the aspirations of individuals based on a uniform distribution. In this case, the aspiration is so personal that it is of great possibility that no one shares the aspiration with anyone else. The numerical results imply that the coefficients are i) exactly the same for both well-mixed populations and rings, ii) σ0 = 1, σ1 = 2 and σ2 = 1 ((Table 3)). Therefore they are in perfect agreement with the theorem and Eq (10).

thumbnail
Table 3. Estimating the coefficients σks by simulation.

We establish three sets of individualised aspirations based on the uniform distribution in [0, 1] (see S2 File for details). For three-player games with b0 = b1 = b2 = 0, the estimated coefficients σ0, σ1, σ2 are obtained by linear regression model in the form of σ0a0 + σ1a1 + σ2a2 + Intercept. For rings, the regression coefficients σ0,σ1,σ2 and Intercept are close to 1,2,1 and 0, which agrees perfectly with theoretical calculations Eq (10). In addition, it holds for all the three sets of aspirations, validating the theorem. For well-mixed populations, the d coefficients still hold, consistent with [34]. Thus the coefficients are robust to the heterogeneity in aspiration for both ring and well-mixed population. The confidence interval for the corresponding estimated coefficients (EC) are [EC-ME, EC+ME], where ME in the parentheses is calculated with confidence level 95%. Please refer to the Methods to see the details of the simulation. Population size N = 100, selection intensity β = 5 × 10−2. More details of the simulation are found in S1 File.

https://doi.org/10.1371/journal.pcbi.1006035.t003

Simulation results, i.e., Fig (2b) and (2c) show that the criterion in the theorem to favor strategy A is also valid for random and scale-free networks, beyond regular networks, i.e., Fig (2a). In particular, simulations also show that the criterion is independent of individual aspirations. It is noteworthy that the number of neighbors each individual has should be no less than d − 1 for non-regular networks, such that everyone has enough neighbours to play the d-player game with.

Up till now, we assume that an individual organizes a d−player game with its d − 1 neighbours, and it gets its final payoff via this single game. We now allow this individual participates in the d−player games organized by thy neighbours besides the one organized by itself. In other words, the focal individual takes part in d such d−player games, one of which is organized by itself and all the rest d − 1 games are organized by its d − 1 neighbors. The final payoff of the focal individual is the average payoff over those d games. In this case, the focal individual’s final payoff is not only determined by its nearest neighbors, but also is up to the second nearest neighbours. We investigate whether this alternative way of payoff collection alters the criterion under which strategy A is more abundant. We perform simulations on rings ((Fig 3)). We find that heterogeneity in aspiration does not change the evolutionary outcome either. Thus, the simulation suggests that the way of payoff accumulation does not change our result that the evolutionary outcome is insensitive to the individualised aspirations.

thumbnail
Fig 3. Robustness of the evolutionary outcome based on the way of payoff accumulation.

Individuals are now participating in three games for a time. One of the three is organised by itself and the rest two by its two nearest neighbours. The final payoff is the average of the accumulated payoff in the three games. This payoff accumulation is different from that in the theorem, where every individual only plays a single d−player game. Yet we find that the heterogeneity of aspiration still does not change the abundance of strategy A. Thus our theorem might be generalised to other ways of payoff accumulation. Details of the simulation: The aspiration values are randomly selected from the uniform distributions on interval [0, 1] (left panel) or [0, 5] (right panel). Within each distribution, we establish two populations with different individualised aspirations. All the rest parameters are the same as those of (Fig 1).

https://doi.org/10.1371/journal.pcbi.1006035.g003

Note that the theorem is obtained under the assumption of weak selection limit. It implies that the selection intensity can be sufficiently small, given the payoff entries and individual aspirations. In other words, given the payoff table and the individualised aspirations, the criterion is valid for ensuring that strategy A is more abundant than strategy B, provided the selection intensity can be tuned as small as possible. This does not mean that given a small yet fixed selection intensity and individualised aspiration, the criterion is valid. In fact, fixed selection intensity violates the assumption of the weak selection limit in our theorem. However, it brings us into a question in simulation: how weak the weak selection is to ensure that the criterion in the theorem is valid for a given individualised aspiration. The key idea is to ensure that perturbation analysis is correct. In other words, all the individuals have to be mainly driven by blind switching. And the switching probability from A to B differs little from the other way around, or the difference should be of order β. In other words, g(β(eiπA)) − g(β(eiπB)), i.e., (11) should be of order β. It is required that makes the criterion valid. In other words, if g′′(0) ≠ 0, a sufficient condition to ensure that the criterion is valid is that

It suggests that the larger the maximum value of the aspiration, the weaker the selection intensity is to ensure that the criterion is valid. This is consistent with Fig 1 that aspirations generated via uniform distribution on the interval [0, 5] need weaker selection intensity to make the criterion valid than those generated via the uniform distribution on the interval [0, 1]. If g′′(0) = 0, then holds. Higher order expansion of the difference has to be performed to figure out how weak the weak selection is.

Discussion

Evolutionary processes are typically categorised into two classes, one is comparison based and the other is self-evaluation based. Comparison means that strategies are adjusted via payoff comparison. For example, imitation is comparison based. Therein, an individual compares its payoff with another one’s, and is likely to adopt the strategy of the more successful. Thus the comparison happens between a pair of individuals. Fitness-based process is also comparison based, including Moran process [3639] (Death-birth and Birth-death) and Wright-Fisher process [40]. Therein, individuals reproduce with a probability, which is based on the payoff comparison among all the competing individuals. Individuals with higher payoffs are more likely to reproduce. In this case, the comparison occurs among all the individuals in the population. Self-evaluation indicates that an individual is likely to alter its strategy if the payoff does not meet its aspiration. It is similar to the ‘Win-stay-lose-shift’ strategy [41]. For the self-evaluation process, we do not take the cost of evaluation into account for the sake of simplicity following the convention in evolutionary game theory [4245]. Compared with the comparison-based individuals, individuals with self-evaluation based updating rules only make use of their own information to update the strategy. In addition it introduces ‘exploration’ [46], since an entire population with all the same strategy can have a new mutant, provided an individual is not satisfied with its current payoff. It, however, is impossible for comparison-based individuals unless exploration is additionally assumed. Differences between the two classes of evolutionary processes suggest that they can lead to different evolutionary outcomes. In fact, differences arise even when the population is well-mixed and selection intensity is weak [34], where evolutionary outcomes are almost identical for all the comparison-based processes [23, 47].

Strategy updating rule, either comparison-based or self-evaluation-based, has been typically assumed to be homogeneous for all the individuals. All the individuals make decisions in the same way. Yet strategy updating rule is a personal trait and differs among individuals. We tackle how the heterogeneity in strategy updating rules alters the evolutionary outcome. Previous studies have shown that this heterogeneity could dramatically alter the evolutionary outcome, if individuals are equipped with personalised comparison-based rules [28, 29, 31]. In particular, the heterogeneity can promote cooperation in spatially structured populations. This is in line with the expectation, since the evolutionary outcome is sensitive to the details of the comparison-based processes, even if the updating rule is homogeneous [18]. However, it is still unclear what happens if the evolutionary processes are chosen in the class of self-evaluation based processes. Or how this heterogeneity affects the evolutionary outcome within the class of self-evaluation processes. Intuitively, it is natural to expect that alternative outcomes emerge if individuals are using their own self-evaluation rules. Our theorem, however, shows the opposite: the outcome based on heterogeneous aspirations is the same as that for homogeneous ones under weak selection limit. In fact, it results from the fact that the evolutionary outcome is robust for different aspiration levels if they are homogeneous [34].

The theorem ensures that the criterion to favor strategy A under weak selection limit is a linear inequality, which is independent of all the aspirations. Based on the theorem, the coefficients of the inequality with heterogeneous aspirations are calculated via that with homogeneous aspirations instead. It paves the way to obtain the criterion analytically based on the mean-field approximation [48] (see section Calculation of coefficients S1 File). Note that the calculation of the coefficients itself has shown that the criterion to favor strategy A does not depend on the aspiration for homogeneous aspiration (see Eq. (12) in S1 File). Yet the calculation for the homogeneous case cannot be extrapolated to the heterogeneous case. The obstacle is overcome by the theorem, which ensures the equivalence of the criterions between homogeneous and heterogeneous aspirations.

The proof of our theorem is similar to the proof in [24, 27, 49] in the sense that they both are based on weak selection limit and symmetric evolutionary rule. Yet a transition probability for the aspiration dynamics is a function of the products of payoff entry and selection intensity and the products of the aspiration and selection intensity, whereas a transition probability for [24, 27, 49] is a function of the products of payoff entry and selection intensity only. This mathematical difference prevents us from directly applying their results [24, 49], and we overcome the obstacle by introducing the two lemmas in the Method section.

We concentrate on a regular network with degree d − 1. It is to ensure that every individual has exactly d − 1 neighbors such that every individual is able to play the d-player game. In fact, the proof of the theorem applies to a set structured population, in which all the individuals are in club-like sets. Therein, each sets consists of d individuals, but the number of sets one belongs to varies from individual to individual. In particular, a regular network with degree d − 1 is a set structured population, whose set size is d and the number of the sets one belongs to is d for all the individuals [50]. Furthermore, Fig 2 shows that the criterion in the theorem also holds for random and scale-free networks across a wide range of distributions of individual aspirations. It suggests that the theorem can be true for a wide range of population structures, which requires investigation in the future.

(Fig 1) indicates that there are games such that the heterogeneity in aspiration does not alter the rank of the abundances for non-weak selection, although our theorem is based on the weak selection limit. It seemingly suggests that the criterion to favor a strategy A under strong selection is the same as that under weak selection. Yet, it is not true. Let us consider a multiplayer game whose payoff entries are of order 1. In this case, the payoff is of order 1 for both strategies. Let us assume that there are two types of aspirations in the population, one is positive and the other is negative, which are both of higher order than 1. Under strong selection, individuals with high aspiration are never satisfied with their payoffs, and they switch their strategies all the time. By the symmetry of the updating rule (similar to the proof in Section The average abundance is 1/2 for vanishing selection intensity in S1 File), the two strategies are equally abundant among individuals with high-aspiration. Yet individuals with low aspiration are always satisfied and they act as zealous individuals who never alter strategies [51, 52]. Thus individuals using strategy A are more abundant eventually if individuals using strategy A with low aspiration are more abundant than those using strategy B with low aspiration in the beginning. It clearly indicates that the criterion under which a strategy is more abundant under strong selection differs from that under weak selection. Therefore there are games such that the ranking of strategies is sensitive to the selection intensity in a population with heterogeneous aspirations. In other words, the criterion in the theorem to favor a strategy in abundance is sensitive to the selection intensity, in line with the mutation-selection dynamics [18, 21].

The criterion under which a strategy is more abundant than the other is robust under aspiration dynamics for weak selection limit. In fact, the criterion holds true for various networks, individualised aspirations, and different ways of payoff collection. A potential way to violate the present criterion is to allow individuals to use different updating functions when using strategy A and B. In this case, updating of the two strategies is not symmetric anymore. And the two strategies can be different in abundance even when all of the payoff entries in the table are the same, violating Lemma 2, which is crucial for the theorem. A study along this way is undergoing.

To sum up, in the limit of weak selection, we show that the heterogeneity in aspiration does not alter the evolutionary outcome compared with the homogeneous case. Our result not only shows a striking difference between comparison based and self-evaluation based rules, but also provides a novel way to compute the aspiration dynamics.

Supporting information

S1 File. Individualised aspiration dynamics.

https://doi.org/10.1371/journal.pcbi.1006035.s001

(PDF)

S2 File. Table of personal aspiration values in simulations.

In the table, personal aspiration values for different simulations are listed. For example, the second column (Column B) shows aspiration values of all the individuals indexed from 1 to 100 and it corresponds to Set 1 of the uniform distribution on the interval [0, 1] used in Figs (1) and (3) (red circles in the left panels).

https://doi.org/10.1371/journal.pcbi.1006035.s002

(XLSX)

Acknowledgments

We appreciate the Levin Lab for insightful discussions.

References

  1. 1. Nowak MA, Sigmund K. Evolutionary Dynamics of Biological Games. Science. 2004;303:793–799. pmid:14764867
  2. 2. Flood M, Dresher M, Tucker A, Device F. Prisoner’s Dilemma: Game Theory. Experimental Economics. 1950;.
  3. 3. Milinski M, Semmann D, Krambeck HJ, Marotzke M. Stabilizing the Earth’s climate is not a losing game: Supporting evidence from public goods experiments. Proceedings of the National Academy of Sciences USA. 2006;103:3994–3998.
  4. 4. Santos FC, Vasconcelos V, Santos MD, Neves P, Pacheco JM. Evolutionary dynamics of climate change under collective-risk dilemmas. Mathematical Models and Methods in Applied Sciences. 2012;22:1140004.
  5. 5. Abou Chakra M, Traulsen A. Evolutionary dynamics of strategic behavior in a collective-risk dilemma. PLoS Computational Biology. 2012;8:e1002652. pmid:22927807
  6. 6. Ratcliff WC, Fankhauser JD, Rogers DW, Greig D, Travisano M. Origins of multicellular evolvability in snowflake yeast. Nature Communications. 2014;6:6102.
  7. 7. Perc M, Gómez-Gardeñes J, Szolnoki A, Floría LM, Moreno Y. Evolutionary dynamics of group interactions on structured populations: A review. Journal of The Royal Society Interface. 2013;10(80).
  8. 8. Helbing D, Szolnoki A, Perc M, Szabó G. Evolutionary establishment of moral and double moral standards through spatial interactions. PLoS Computational Biology. 2010;6:e1000758. pmid:20454464
  9. 9. Hauert C. Spatial effects in social dilemmas. Journal of Theoretical Biology. 2006;240:627–636. pmid:16352316
  10. 10. Fu F, Chen X, Liu L, Wang L. Social dilemmas in an online social network: the structure and evolution of cooperation. Physics Letters A. 2007;371(1):58–64.
  11. 11. Ohtsuki H, Iwasa Y. The leading eight: Social norms that can maintain cooperation by indirect reciprocity. Journal of Theoretical Biology. 2006;239:435–444. pmid:16174521
  12. 12. Masuda N, Ohtsuki H. Tag-based indirect reciprocity by incomplete social information. Proceedings of the Royal Society B. 2007;274:689–695. pmid:17254993
  13. 13. Ohtsuki H, Hauert C, Lieberman E, Nowak MA. A simple rule for the evolution of cooperation on graphs. Nature. 2006;441:502–505. pmid:16724065
  14. 14. Zhou L, Li A, Wang L. Evolution of cooperation on complex networks with synergistic and discounted group interactions. Europhysics Letters. 2015;110(60006).
  15. 15. Li A, Wu B, Wang L. Cooperation with both synergistic and local interactions can be worse than each alone. Scientific Reports. 2014;4 (5536).
  16. 16. Maciejewski W, Fu F, Hauert C. Evolutionary game dynamics in populations with heterogeneous structures. PLoS Computational Biology. 2014;10:e1003567. pmid:24762474
  17. 17. Wu B, Bauer B, Galla T, Traulsen A. Fitness-based models and pairwise comparison models of evolutionary games are typically different—even in unstructured populations. New Journal of Physics. 2015;17:023043.
  18. 18. Wu B, García J, Hauert C, Traulsen A. Extrapolating weak selection in evolutionary games. PLoS Computational Biology. 2013;9:e1003381. pmid:24339769
  19. 19. Grüter C, Czaczkes T, Ratnieks F. Decision making in ant foragers (Lasius niger) facing conflicting private and social information. Behavioral Ecology and Sociobiology. 2011;65:141?? 148.
  20. 20. Pfeffer SE, Wittlinger M. Optic flow odometry operates independently of stride integration in carried ants. Science. 2016;353(6304):1155–1157. pmid:27609893
  21. 21. Roca CP, Cuesta JA, Sanchez A. Evolutionary game theory: Temporal and spatial effects beyond replicator dynamics. Physics of Life Reviews. 2009;6(4):208–249. pmid:20416850
  22. 22. Gokhale CS, Traulsen A. Evolutionary games in the multiverse. Proceedings of the National Academy of Sciences USA. 2010;107:5500–5504.
  23. 23. Wu B, Altrock PM, Wang L, Traulsen A. Universality of weak selection. Physical Review E. 2010;82:046106.
  24. 24. Tarnita CE, Ohtsuki H, Antal T, Fu F, Nowak MA. Strategy selection in structured populations. Journal of Theoretical Biology. 2009;259:570–581. pmid:19358858
  25. 25. Kimura M, Crow JF. The number of alleles that can be maintained in a finite population. Genetics. 1964;49:725–738. pmid:14156929
  26. 26. Wu B, Traulsen A, Gokhale CS. Dynamic properties of evolutionary multi-player games in finite populations. Games. 2013;4(2):182–199.
  27. 27. Nathanson CG, Tarnita CE, Nowak MA. Calculating evolutionary dynamics in structured populations. PLoS Computational Biology. 2009;5(12):e1000615. pmid:20019806
  28. 28. Pinheiro FL, Santos FC, Pacheco JM. How selection pressure changes the nature of social dilemmas in structured populations. New Journal of Physics. 2012;14(7):073035.
  29. 29. Rong Z, Wu Z, Hao D, Chen MZQ, Zhou T. Diversity of timescale promotes the maintenance of extortioners in a spatial prisoner’s dilemma game. New Journal of Physics. 2015;17(3):033032.
  30. 30. Santos FC, Pinheiro FL, Lenaerts T, Pacheco JM. The role of diversity in the evolution of cooperation. Journal of Theoretical Biology. 2012;299:88–96. pmid:21930134
  31. 31. Perc M, Szolnoki A. Social diversity and promotion of cooperation in the spatial prisoner’s dilemma game. Physical Review E. 2007;77(1):011904.
  32. 32. Szolnoki A, Perc M, Szabo G. Diversity of reproduction rate supports cooperation in the prisoner’s dilemma game on complex networks. European Physical Journal B. 2008;61(4):505–509.
  33. 33. Santos FC, Santos MD, Pacheco JM. Social diversity promotes the emergence of cooperation in public goods games. Nature. 2008;454:213–216. pmid:18615084
  34. 34. Du J, Wu B, Altrock PM, Wang L. Aspiration dynamics of multi-player games in finite populations. Journal of Royal Society Interface. 2014;11(94):1742–5662.
  35. 35. Allen B, Lippner G, Chen YT, Fotouhi B, Momeni N, Yau ST, et al. Evolutionary dynamics on any population structure. Nature. 2017;544:227–230. pmid:28355181
  36. 36. Moran PAP. The Statistical Processes of Evolutionary Theory. Clarendon Press, Oxford; 1962.
  37. 37. Traulsen A, Nowak MA, Pacheco JM. Stochastic Dynamics of Invasion and Fixation. Physical Review E. 2006;74:011909.
  38. 38. Blume LE. The statistical mechanics of best-response strategy revision. Games and Economic Behavior. 1995;11(2):111–145.
  39. 39. Altrock PM, Traulsen A. Deterministic evolutionary game dynamics in finite populations. Physical Review E. 2009;80:011909.
  40. 40. Imhof LA, Nowak MA. Evolutionary game dynamics in a Wright-Fisher process. Journal of Mathematical Biology. 2006;52:667–681. pmid:16463183
  41. 41. Nowak MA, Sigmund K. A strategy of win-stay, lose-shift that outperforms tit-for-tat in the Prisoner’s Dilemma game. Nature. 1993;364:56–58. pmid:8316296
  42. 42. Du J, Wu B, Wang L. Aspiration dynamics and the sustainability of resources in the public goods dilemma. Physics Letters A. 2016;380(16):1432–1436.
  43. 43. Perc M, Wang Z. Heterogeneous aspirations promote cooperation in the prisoner’s dilemma game. PLoS One. 2010;5(12):e15117. pmid:21151898
  44. 44. Chen X, Wang L. Promotion of cooperation induced by appropriate payoff aspirations in a small-world networked game. Phys Rev E. 2008;77(017103).
  45. 45. Oechssler J. Cooperation as a result of learning with aspiration levels. Journal of Economic Behavior & Organization. 2002;49:405–409.
  46. 46. Traulsen A, Hauert C, De Silva H, Nowak MA, Sigmund K. Exploration dynamics in evolutionary games. Proceedings of the National Academy of Sciences USA. 2009;106:709–712.
  47. 47. Nowak MA, Sasaki A, Taylor C, Fudenberg D. Emergence of cooperation and evolutionary stability in finite populations. Nature. 2004;428:646–650. pmid:15071593
  48. 48. Du J, Wu B, Wang L. Aspiration dynamics in structured population acts as if in a well-mixed one. Scientific Reports. 2015;5 (8014).
  49. 49. Tarnita CE, Wage N, Nowak MA. Multiple strategies in structured populations. Proceedings of the National Academy of Sciences USA. 2011;108:2334–2337.
  50. 50. Wu B, Arranz J, Du J, Zhou D, Traulsen A. Evolving synergetic interactions. Journal of The Royal Society Interface. 2016;13.
  51. 51. Coleman JS. Free Riders and Zealots: The Role of Social Networks. Sociological Theory. 1988;6(1):52–57.
  52. 52. Masuda N. Evolution of cooperation driven by zealots. Scientific Reports. 2012;2(646).