## Abstract

Darwinian dynamics based on mutation and selection form the core of mathematical models for adaptation and coevolution of biological populations. The evolutionary outcome is often not a fitness-maximizing equilibrium but can include oscillations and chaos. For studying frequency-dependent selection, game-theoretic arguments are more appropriate than optimization algorithms. Replicator and adaptive dynamics describe short- and long-term evolution in phenotype space and have found applications ranging from animal behavior and ecology to speciation, macroevolution, and human language. Evolutionary game theory is an essential component of a mathematical and computational approach to biology.

Evolution through natural selection is often understood to imply improvement and progress. A heritable trait that confers to its bearer a higher fitness will spread within the population. The average fitness of the population would therefore be expected to increase over time. This is often pictured as a steady ascent on a so-called fitness landscape. The landscape metaphor suggests some solid ground over which the population moves. This paradigm (*1*), which is also widespread in the theory of genetic algorithms (*2*), neglects one-half of the evolutionary mechanism: Although the environment selects the adaptations, these adaptations can shape the environment. By moving across a fitness landscape, populations change that landscape (Fig. 1).

This is particularly clear if several populations interact, because each population can be part of the fitness landscape of the other. A host's successful immune response to a pathogen, for instance, will exert selection pressure leading to adapted strains of pathogens, and vice versa (*3*–*5*). But even within a single population, the fitness of a trait often depends on the prevalence of that trait: The selective advantage of a given tree height, for example, depends on the heights of neighboring trees. Similarly, the success of a given sex ratio depends on the overall sex ratio in the population.

Therefore, the fitness landscape is shaped by the phenotypic distributions of the involved populations. As the population moves through the fitness landscape, new peaks and valleys form, channeling its further motion. This viewpoint affects not only the intuition of evolutionary biologists but also their theoretical tools. The proper technique for describing uphill motion on solid ground is optimization theory, a set of mathematical techniques developed in the past 300 years, mostly to solve physical or technical problems. If the adaptive steps, however, imply changes in the environment, eventually necessitating new adaptations, then game theory is the appropriate framework. This technique originated more than 50 years ago to tackle economic and social problems involving interdependencies among several agents. Evolutionary biologists soon understood its potential and started applying it to evolutionary problems (*6*–*8*). The success of a strategy in a game depends on the co-player's strategy, much as the fitness of a phenotype depends on the composition of the population. Roughly speaking, game theory is the mathematical toolbox for methodological individualism, the systematic attempt to found social theory on the actions and needs of individual agents (*9*, *10*). For outcomes shaped by “selfish genes” or by the selfish “homo economicus,” this is the proper instrument.

## Biological Games

The number of papers applying game theory to biological problems is in the thousands (*11*). The first use of game-theoretic arguments can be found in the field of sex-ratio theory (*12*, *13*). Evolutionarily stable strategies (ESSs) were introduced to discuss the prevalence of ritual fighting in interspecific animal conflicts, which also led to asymmetric and multistep games (*7*, *14*). Evolutions of root structure or tree height are problems of resource allocation (*15*). Conflicts concerning mate choice (*16*), sibling rivalry (*17*), and parent-offspring antagonism (*18*) are a rich mine of game-theoretic models; so are social foraging, dispersal, and habitat selection (*19*). The arms races between predators and prey, or between parasites and their hosts, offer many examples of games between distinct populations (*20*). Communication in its widest sense, including alarm calls, threat displays, or sexual advertisement, lead to game-theoretic problems concerning bluff and honest signaling (*21*). Acquisition and performance of human language in a heterogeneous population can be studied as an evolutionary game (*22*). Increasingly, evolutionary game theory is used in economic and social sciences and applied to experimental games with human subjects (*23*–*27*). Even genes, bacteria, organelles, and viruses can be engaged in games of cooperation and conflict (*28*–*31*).

The evolution of virulence of infectious agents is another vast field that makes use of game-theoretic arguments. The classical understanding, based on constant selection, is that parasites evolve to maximize their basic reproductive ratio (*32*). Frequency-dependent selection arises when several parasite strains superinfect the same host or when rapid evolution generates many different parasite mutants in any one infected individual (*33*). Lack of cooperation among parasites can lead to short-sighted, maladapted levels of excessive virulence harming both host and parasite.

The growth in the range of applications demanded an extension of classical game theory, away from the prevalent static doctrine dominated by the equilibrium notion of Nash and by the quest for a “unique solution” to rational play. The concepts of “unbeatable strategy” (*6*) and “evolutionary stability” (*7*) implicitly assumed some underlying population dynamics describing the potential success of invading mutants and, more generally, the interplay of mutation with frequency-dependent selection. An exact formulation of these population dynamics depends on the structure of the population (for instance, well-mixed or sessile), on the mechanisms for the transmission of the relevant traits (by genetic inheritance or cultural learning), and on the time scales underlying the evolutionary and ecological processes. Remarkably, for each of the plethora of conceivable adjustment dynamics (*34*), there exist simple games (variants of “rock-scissors-paper”) where the long-term outcome is not a Nash equilibrium but endless regular or irregular oscillations: Hence, the static approach is in principle unable to provide a full analysis. Only a dynamical theory can describe the “Red Queen” phenomena (*35*) that are prevalent in evolution, for example, in hostparasite interactions, in the arms races between predators and prey, or in fluctuating degrees of cooperation (*36*).

Before reviewing the recent developments in evolutionary game theory, we sketch a few basic types of interactions that help to familiarize readers with terminology. A “game” is an interaction between a set of individuals. These “players” act according to their behavioral phenotypes, which are called “strategies.” The players' payoffs, which translate into fitness, depend, in general, on their own strategy and on that of their co-players. A tree's height, a parent's sex ratio, a parasite's virulence, a female's choosiness, or a male's ornament are instances of strategies. This terminology is by now well established, but occasionally still induces reactions like “animals don't play games.”

## Strategic Interactions and Population Structures

The conceptually simplest games offer only two strategies and four outcomes. We illustrate this with the example of some cooperative interaction between two players. Each has the possibility to cooperate, C, or to defect, D. If both play C, both obtain a higher payoff than if both play D, but if the coplayer cooperates, it is better to defect and to exploit the other's efforts. Two alternatives can hold if the coplayer defects: (i) it could be best also to defect, in order not to get exploited; this case represents the well-known prisoner's dilemma game (*37*); or (ii) it could be best to cooperate nevertheless, which leads to the snowdrift game (*38*). If two drivers are caught with their cars in a snowdrift and one of them refuses to cooperate in shoveling a way out, the other driver is better off to cooperate unilaterally, rather than spend the night freezing. In the prisoner's dilemma game, D is a best reply no matter whether the co-player uses C or D. In the snowdrift game, each strategy is a best reply to the other.

The snowdrift game is also known as hawk-dove or chicken game. The first name comes from a situation where animals fight for a territory: “Hawks” escalate the fight, risking serious injury, whereas “doves” flee when the opponent escalates. In the chicken game, two cars are heading for a collision. The loser chickens out, while the winner stays on track. Big-time loss occurs when both stay on track.

A major challenge in experimental or observational studies is to determine the ranking of payoff values. For example, the interaction of female lions defending a territory against invaders has been interpreted as a prisoner's dilemma (*39*). But, the observation that cooperating and defecting lionesses happily coexist in a group makes it more likely that these lions play chicken.

The payoff values of a prisoner's dilemma have been measured for selection between two mutants of the bacteriophage Φ6. The cooperator builds large amounts of products required for reproduction, whereas the defector specializes in the use of these products when both mutants are in the same cell (*40*, *41*).

A biochemical example for the prisoner's dilemma is provided by the evolution of adenosine triphosphate (ATP)–producing pathways. Cooperators have a low rate but high yield of ATP production, whereas defectors have a high rate but low yield. The resulting game could have played a major role in the emergence of multicellularity (*29*).

In general, evolutionary dynamics of two strategies, A and B, have four outcomes (Fig. 2A). (i) Dominance: A vanishes, if B is the best reply to both A and B. (ii) Bistability: Either A or B vanishes, depending on the initial mixture, if each strategy is the best reply to itself. (iii) Coexistence: A and B coexist in stable proportions, if each strategy is the best reply to the other. (iv) Neutrality: The frequencies of A and B are only subject to random drift, if each strategy fares as well as the other for any composition of the population. The former three cases correspond to the familiar ecological scenarios of two-species competition (*42*).

Examples for all four cases can be found in the repeated prisoner's dilemma, where an interaction between two players consists of many rounds. Tit-for-tat (TFT) is a strategy which cooperates in the first round and then repeats whatever the other player did in the previous round. “Always defect” (AllD) is bistable with TFT if the average number of rounds is sufficiently high. “Always cooperate” (AllC) and TFT are neutral if there is no noise and can coexist in the presence of noise. AllC is dominated by AllD.

With three strategies, the outcome can be considerably more complicated, mainly because of the possibility of “rock-scissors-paper” cycles: It may happen that B dominates A, C dominates B, and A, in turn, dominates C (Fig. 2B). In this case, the dynamics can lead, depending on the relative strengths of the interactions, to either longterm coexistence of all three strategies or to ever-increasing oscillations ending, eventually, with the elimination of two strategies. Examples of such situations, originally viewed as theoretical issues only, have recently been found in nature. (i) There exist three morphs of the male lizard *Uta stansburiana* who differ in their throat color and in their mate-guarding behavior. Type A is monogamous and succeeds in preventing other males from approaching their mate. Type B is polygamous and less efficient, having to split its efforts on several females. Type C does not engage in female-guarding behavior at all but roams around in search of sneaky matings (*43*). (ii) There exist three strains of *Escherichia coli* bacteria. Type A releases toxic colicin and produces, for its own protection, an immunity protein. Type B produces the immunity protein only. Type C produces neither toxin nor immunity (*44*).

We have implicitly assumed so far that interactions within a population are on the basis of random encounters. In many important situations, however, such well-mixing cannot be assumed, and the population structure affects the outcome substantially. If cooperators, for example, preferentially assort with other cooperators, they need not be outcompeted by defectors. Such preferential assortment can be achieved, for instance, if players mostly interact with close relatives (*45*). This raises the issue of kin selection and, more generally, group selection: In groups with many cooperators, average fitness will be higher (*46*). If the degree of positive assortment in the formation of groups is sufficiently high, then cooperators will not be eliminated, although within each group they do worse than defectors. The basic theoretical tool for investigating such “viscous” populations is the Price equation, which describes the growth rate of a strategy as the sum of two terms denoting selection within and between groups.

In a similar way, the outcome of strategic interactions can be affected by preferential assortment with close neighbors. If games are played between neighbors and offspring move only to adjacent sites, then dominated strategies need not be eliminated. The best-known example occurs for the prisoner's dilemma, where unconditional cooperators can subsist, often in frequencies and patterns wildly oscillating in space and time (*47*, *48*) (Fig. 3). A lab experiment has recently highlighted the effect of spatial structure on the population dynamics of the three aforementioned *E. coli* strains engaged in rock-scissors-paper competition. If they live in a well-mixed flask and the resulting population is used to seed, after a few bacterial generations, another flask, then such serial transfers will eventually lead to the survival of a single strain only. If, however, the bacteria grow on the surface of an agar plate and a two-dimensional sample of this surface is used to seed the next agar plate, without altering the neighborhood structure, then the resulting sequence of serial transfers will preserve all three strains (*44*). Often, spatial structure tends to allow more diversity than prevails in well-mixed populations.

## Replicator Dynamics and Short-Term Evolution

What are the dynamical systems used to analyze frequency-dependent selection for biological games? A standard tool is the replicator equation (*49*–*51*) (Fig. 4). It assumes a well-mixed population with a finite number of strategies and posits that the per capita growth rate of each strategy is proportional to its fitness. If the fitness values are independent of the frequencies, then the average fitness of the population will grow: This yields the usual hill-climbing dynamics and results in the survival of the fittest only. But if the fitness values are frequency-dependent, as is usual with evolutionary games, the average fitness need not grow (*1*). Several strategies can coexist in steady or fluctuating frequencies. Chaotic dynamics are possible.

Many examples of replicator dynamics use linear fitness functions, which is the case if payoff results from pairwise encounters. Games can also be played among larger groups, for instance, public goods games (*52*), or even within the entire population, as in the sex-ratio game. In this case, fitness is usually a nonlinear function of the frequencies.

Replicator dynamics have their basis in the assumption of clonal, genetic reproduction, but these dynamics and variants of it can also describe the spread of strategies by nongenetic mechanisms such as learning, imitation, or other forms of cultural evolution (*50*, *53*, *54*).

This array of dynamical systems has to be modified if spatial structure is included in the model. If the population is distributed in a one- or two-dimensional continuum, for instance, then reaction-diffusion equations can describe the spread of strategies via traveling waves or the formation of stable spatial patterns (*55*). If players are territorial, some models typically assume that they are located on a grid-like structure and interact with their nearest neighbors only (*47*). Players adopt the strategy of whichever neighbor obtained the highest total payoff. Again, many variants are possible in terms of lattice geometry, neighborhood structures, and whether updating is synchronous or not. Cellular automata of this type have a rich dynamic behavior, possibly as complex as universal Turing machines. The relationship between spatial models and “mean-field” or “pair-approximations” offer many challenges of mathematical and computational nature (*56*, *57*).

Replicator dynamics deal with relative abundances (equal to frequencies). If fitness values depend in addition on the total population size, as is natural in many ecological contexts, then the dynamics have to be supplemented with an equation for the total population size or replaced by equations for absolute abundances (*58*).

## Adaptive Dynamics and Long-Term Evolution

The replicator equation describes selection only, no drift and no mutation. A strategy missing in the initial population remains absent. The dynamics can be modified by adding a small steady rate of miscopying. The resulting outcome will depend on the relative sizes of the mutation rates. A more widespread approach in evolutionary games is to investigate the impact of mutations without explicitly modeling their origin. The new strategy is introduced with an initial frequency that is very small (so that it does not affect the fitness of the residents). Will that frequency decrease or not? Will the mutant be eliminated under the effect of selection, or will it invade the population? To return to the case of two types A and B only: Dominance holds if A can invade B and B cannot invade A (or vice versa), coexistence holds if A can invade B and B can invade A, and bistability holds if neither type can invade the other.

More generally, given a resident population with its set of strategies, one can study the fate of an additional strategy by considering the replicator dynamics in the augmented space (resident and mutant population) and computing the growth rate of the mutant for small values. If this growth rate, the “Darwinian fitness” of the mutant, is positive, then the mutant will invade. In more complex situations, if the resident population is not at equilibrium, the population is age-structured, or the environment is stochastic, the fitness definition must be adapted (*59*). A successful invasion by itself tells us nothing about the subsequent fate of the mutant. It is possible that (i) it gets added to the resident cast, so that the new population contains one more strategy, (ii) eliminates one or several of the resident strategies, or (iii) eventually gets eliminated itself in a “resident strikes back” reaction (*60*).

The invasibility issue forms the core of evolutionary game theory. A strategy is said to be evolutionarily stable if a resident population cannot be invaded by a mutant strategy under the influence of natural selection, as long as its initial frequency is sufficiently small (*7*). Checking for this condition requires the specification of the range of possible mutants. In the special case that the resident actually dominates all mutants, the strategy is said to be “unbeatable” (*6*). The demands on an ESS are weaker: The mutant must either be dominated or else form a bistable pair with the resident strategy. In this latter case, if the initial frequency of the mutant exceeds a so-called “invasion barrier,” it can spread and eventually eliminate the resident. Both stability concepts are variants of the game-theoretic notion of a Nash equilibrium, meaning in this context a strategy with the property that, if all players adopt it, none has an incentive to deviate unilaterally. Whereas Nash equilibria exist for every game, this is not the case for unbeatable or ESSs. It should also be stressed that such strategies are not necessarily optimum: It can be that all players would be better off if they jointly deviated, in a correlated way, with the use of another strategy (for example, cooperation in the prisoner's dilemma). Such a concerted action is beyond the means of natural selection, but an evolutionarily stable population can conceivably be invaded simultaneously by two or more mutant strategies (*61*).

The theoretical approach through adaptive dynamics considers the opposite, “mutationlimited” scenario: Only one mutant enters at a time, and it is either driven to extinction or fixed in the population before the next mutant occurs (Fig. 5). Thus, mutational steps are supposed to occur only rarely, compared with the time needed to substitute the resident with the mutant, and the population is essentially always monomorphic. The successful invasion attempts lead to a substitution sequence in trait space, that is, the space of all possible strategies, not only those actually present within the population. This phenotypic space is in general a continuum (of possible tree heights, sex ratios, or ages of maturation) and can be multi- and even infinite-dimensional.

If the mutant phenotype is always very close to the resident's, then the discrete-step substitution sequence can be approximated by a smooth orbit of the so-called adaptive dynamics in trait space (*62*–*64*), pointing into the direction of the most favorable among all possible closeby traits. Whereas the space of replicator dynamics consists of the frequencies of finitely many phenotypes, adaptive dynamics along the local fitness gradient describes evolution in the continuous space of all possible phenotypes. The price for the richer strategy space consists in assuming that the residents are all using the same strategy.

With adaptive dynamics, the motivation tends to shift from behavioral aspects to more ecological issues. Both evolutionary and adaptive dynamics can be used to model the interaction of several populations. The growth rates of the different strategies will depend, in general, on their frequencies within these populations and on the densities of the populations. The usual, ecological models, for instance, for predator-prey interactions, are limiting cases obtained by assuming that the evolution within each population occurs at a much slower rate than the population regulation. This assumption has recently been challenged in several case studies of rapid evolution showing that population cycles can be driven by evolutionary factors (*65*).

Adaptive dynamics need not lead to an evolutionarily stable state. Cycling can continue forever (*66*). Moreover, if adaptive dynamics converge to an equilibrium, this need not be an ESS. An attractor of adaptive dynamics is said to be a convergent stable strategy (*67*, *68*). Conversely, an evolutionarily stable equilibrium can be unattainable (*67*, *69*, *70*). In that case, if the resident population is at the equilibrium, fitness is maximum and no mutant can invade, but if the resident population is close to the equilibrium, then only mutants can invade that are less close by. Figure 6 shows as an example adaptive dynamics of reactive strategies in the repeated prisoner's dilemma.

So-called pairwise invasibility plots offer a geometric tool for analyzing mutation-limited evolution (*71*). In particular, they show that adaptive dynamics can lead to a fitness minimum, where the monomorphic population becomes unstable and has to split up (*72*, *19*, *73*). Such evolutionary branching points offer a tempting analogy to sympatric speciation.

A much-studied example involves competition for a common resource, for instance, birds feeding on a type of seed whose size is normally distributed. Without frequency dependence, the birds would be selected to specialize on the most common size of the seed. This specialization, however, might alter the distribution of seed size. Hence, frequency-dependent “disruptive” selection can lead to character divergence, one type specializing on smaller, the other on larger, seeds (*68*, *71*).

## Evolutionary Games and Population Genetics

Whereas the combination of game-theoretic and ecological models offers no conceptual problems, game theory and population genetics mix less readily. This seems paradoxical: The selection equation in population genetics is a replicator equation, and the alleles are perfect examples of replicators, obeying clonal inheritance. Evolutionary game theory, however, describes evolution in phenotype space. In spite of recent progress in genomics and proteomics, little is known about the genotype-phenotype mapping from a discrete allele space to a continuous trait space.

Under appropriate restrictions (if, for instance, trait space is one-dimensional or if only one locus and at most three strategies are involved), a phenotypic ESS can be viewed as stable end point of a mutation-limited genetic evolution. With recombination between several loci, however, frequency-dependent genetics cannot be fully covered by an invasion analysis at equilibrium points (*74*–*76*). Genetic constraints as well as linkage and epistasis introduce formidable analytical challenges. Furthermore, the assumption of a monomorphic resident population is obviously a critical issue.

Nevertheless, adaptive dynamics often allow a good understanding of the coevolution of traits under frequency-dependent selection. In particular, branching points of adaptive dynamics may be at the origin of sympatric speciation. In the case of adaptations to different seed sizes, for instance, an interbreeding population can converge to the branching point, and then a mutation for assortative mating based on the trait can invade, eventually splitting the population into two species diverging apart. Speciation through ecological character displacement has long intrigued geneticists (*77*). Numerical simulations show that genes for assortative mating can even spread if they are based on “markers” having nothing to do with the trait value (*78*); on the other hand, assortative mating could fail to exceed an intermediate level if the corresponding genes have small effects only (*79*).

Such examples show that adaptive dynamics can address macroevolutionary issues by analyzing frequency-dependent selection in ecology-driven models. This allows us to study a wide range of nonstatic scenarios, including Red Queen types of cycling and ecological suicide (*80*), which are far removed from the static outcomes suggested by the Panglossian paradigm of hill-climbing in fitness landscapes. Mathematical and computational approaches are increasingly using game theory, rather than optimization, to model the strategic intricacies of coevolution.

## Looking Ahead

Evolutionary biology is well grounded in mathematical theory. The way populations change under the influence of mutation and selection can be described by a rich array of mathematical equations. Ideas of evolutionary mechanisms must be formulated in terms of mathematical equations for consistent analysis and meaningful investigation. Most often, Darwinian fitness depends on the relative abundance of individual phenotypes within populations, and therefore game theory is the appropriate mathematical tool.

Many challenges lie ahead. Evolutionary game theory is formulated in terms of phenotypes, thereby ignoring the complexity of the genotype-phenotype mapping. More work is needed on the interaction of strategies encoded in genomic sequences. Most evolutionary game dynamics have been studied in the context of infinitely large populations. We expect that finite population size effects will lead to surprising outcomes and might question the importance of traditional evolutionary stability. Cultural interpretations of replicator dynamics often assume that successful strategies spread by imitation or learning, but the learning of complicated strategies from behavioral observations is a nontrivial task that needs specific investigation. Similarly, studying human language requires a connection between the mathematics of game theory, learning theory, and computational linguistics.

Emerging fields as diverse as metabolic control networks within cells and evolutionary psychology, for example, should benefit from game theory. Metabolic networks respond to incoming signals from other cells and in turn generate outgoing messages, thereby representing a grammar of cellular communication. The performance (fitness) of a particular network depends on networks in other cells, and hence frequency-dependent selection is operating. The main themes of evolutionary psychology include cooperation and communication among individuals and are therefore intrinsically game theoretic. Having a theory of mind, for example, is itself a useful strategy that must help winning certain games.

The applications of evolutionary game theory pervade by now all areas of biology. Interactions among genes, viruses, cells, and humans are often instances of evolutionary games that are amenable to empirical and theoretical investigation. Game theory is the appropriate tool whenever the success of an individual depends on others.