Special Reviews

# Decision Theory: What "Should" the Nervous System Do?

See allHide authors and affiliations

Science  26 Oct 2007:
Vol. 318, Issue 5850, pp. 606-610
DOI: 10.1126/science.1142998

## Abstract

The purpose of our nervous system is to allow us to successfully interact with our environment. This normative idea is formalized by decision theory that defines which choices would be most beneficial. We live in an uncertain world, and each decision may have many possible outcomes; choosing the best decision is thus complicated. Bayesian decision theory formalizes these problems in the presence of uncertainty and often provides compact models that predict observed behavior. With its elegant formalization of the problems faced by the nervous system, it promises to become a major inspiration for studies in neuroscience.

Evolutionary psychology has found that many human behaviors can be well understood assuming adaptation of psychology to the past social environment of humans [e.g., (1)]. Similarly, ethology, the study of animal behavior [e.g., (2)], has shown that many of the properties of the nervous system and the bodies of animals are remarkably well adapted to their ecological niche. These disciplines have shown that, over the course of evolution, animals are often endowed with solutions to common problems that are close to optimal [(1), but see (3)]. Many studies in neuroscience analyze low-level processes. For example, researchers study how animals control their limbs, how they infer events in the world, and how they choose one of several possible rewards. Such processes may have remained conserved for very long periods of time. We can thus expect the solution used by the nervous system for such problems to be close to optimal.

Normative models formalize how the idea of adaptation predicts properties of the nervous system. These models assume that a process has an objective (e.g., walk using least energy). Such an objective is typically formalized by a function (“utility”) that defines how good the solution is (e.g., the energy used). By combining the utility function with constraints (e.g., properties of muscles), it is possible to derive the best possible solution to the problem, usually using computer simulations. If the prediction of the model is matched by biology, it is concluded that, indeed, we have understood the purpose of a system. Normative models thus ask the “why?” question and formalize the ultimate purpose of a system. In this review, I focus on normative approaches to decision making; the nervous system often comes up with near-optimal decisions in a world characterized by uncertainty. To understand the nervous system, descriptive knowledge of how the nervous system works should be combined with normative knowledge of which problems it solves.

## Decision Theory

The purpose of the central nervous system is to make decisions so that we can thrive by interacting successfully with our environment (Fig. 1A). When we play darts, we need to decide which position on the dartboard to aim at (Fig. 1B). The dartboard assigns a score to any possible dart position and thus defines the outcome. The objective of playing darts is to obtain as many points as possible. Within decision theory, such an objective is called a utility function [e.g., (4)]. A utility function U(outcome) measures how good or bad any possible decision outcome is. If dart players could choose where the dart will hit the board, they would choose the position that yields the most points and would thus maximize utility.

Although we can freely make decisions, we cannot directly choose the decision outcomes. If we always aim for the same position a, say the center of the bull's eye, and throw many darts, we will produce a distribution of dart positions, x, on the dart board (Fig. 1C, inset). Within decision theory, this probability distribution is denoted p(outcome = x|decision = a). If we aim at the position on the board that gives the highest score, we may instead hit a neighboring area of the dartboard and receive a low score. Depending on the position we aim at, different scores become more or less likely. This is a special case of a general problem in decision theory: Outcomes depend on decisions in a probabilistic way.

To derive the most beneficial decision, it is necessary to combine the utility function with knowledge of how our decisions affect potential outcomes. The expected utility is $$mathtex$$$\ E[Utility(decision)]{\equiv}\$$$mathtex$$ $$mathtex$$$\ {{\sum}_{\begin{array}{c}\mathrm{possible}\\\mathrm{outcomes}\end{array}}}p(outcome|decision)U(outcome)\$$$mathtex$$ The best decision is then defined as the one that will maximize the expected utility (5).

The decision theoretic approach can be used whenever we know how decisions are related to outcomes and we know the utility of the outcomes. We can apply this framework to the example of darts playing. If we had low motor noise, we would be best off aiming at the triple 20 (Fig. 1C). In contrast, if we have a realistic value of motor noise, the best point at which to aim is not the point of maximal score but is to the lower left of the board (Fig. 1D, marked by dart). Both the behavior of advanced amateur players who have moderate motor variance and the behavior of professional players who have low motor variance are predicted by this decision theoretic approach. A range of recent studies of decision-making have analyzed situations that are analogous to playing darts (6). Such simple decisions are well predicted from the assumption that people solve their decision problems in a fashion that is close to optimal. The approach also applies to many animal behaviors: Animals need to choose to forage or rest, fight or flight, continue moving or freeze in place. Moreover, any behavior of an animal is, in some abstract way, a decision. The nervous system chooses one behavior from the set of all possible behaviors. Decision theory is thus a fundamental formalization of many problems that are solved by the nervous system and studied in neuroscience.

A number of recent studies, under the umbrella name of neuroeconomics, have started to analyze how the nervous system represents and computes with a utility function (79). Because utility is central to any decision, it is important to understand how the nervous system represents reward. Utilities are important for the nervous system because it uses them to act successfully. It has been shown that the nervous system represents changes in expected utility in a way predicted by a major learning theory, reinforcement learning (10). Traditionally simple rewards, such as monetary or food incentives, are used in experiments that analyze how the nervous system represents utilities. However, insights from evolutionary biology (1) predict that many different factors will influence the utility of decisions. Indeed, it has been shown that the nervous system exhibits a reward signal when someone else who cheated in a game is punished (11). Future research will have to uncover the full complexity of how utility functions are represented and used by the nervous system.

## Bayesian Statistics

Deciding seems easy: Choose the action that is associated with the highest expected utility. However, the probability of an outcome given the decision is difficult to estimate. To do so, we need to predict how the world will change until the outcome and how any decision would affect the world. This prediction can only be probabilistic because we have uncertainty about the properties of the world, stemming, for example, from noisy perception (12).

Bayesian statistics defines how uncertain pieces of information may be combined into a joint estimate. New information (called a likelihood) needs to be combined or integrated with information from the past (called a prior). Similar problems occur when information from several cues, for example, proprioceptive and visual, needs to be combined into a joint estimate. Bayesian decision theory (13), the use of Bayesian statistics in a decision framework, defines how our beliefs should be combined with our utility function. Because most if not all of our decisions are made in the presence of uncertainty, understanding the way the nervous system deals with uncertainty is central to understanding its normal mode of operation.

Integration of priors and likelihoods. To calculate the probabilities of outcomes, it is often necessary to update our belief from the past (prior) with new knowledge (likelihood). For example, when we play tennis it is helpful to estimate where the ball will land. The visual system, although noisy, still provides us with an estimate or a likelihood of where the ball will land (sketched in red in Fig. 2A). This knowledge may be combined with information obtained from experience; the positions where the ball may land are not uniformly distributed over the court. The locations may be clustered near the boundary lines, where it is most difficult to return the ball. This distribution of positions is called the prior (sketched in green in Fig. 2A). Bayes's rule states that how the probability of the ball landing at position x given our observation o (posterior) needs to be estimated as $$mathtex$$$\ {p(x|o)_{\mathrm{posterior}}}={p(x)_{\mathrm{prior}}}{p(o|x)_{\mathrm{likelihood}}}{/}p(o)\$$$mathtex$$

Recent studies have analyzed such combinations in simple integration problems. Sensorimotor integration, force estimation, timing estimation, speed estimation, the interpretation of visual scenes, just to name a few, have been analyzed (14, 15). Together, these studies demonstrate that people intuitively combine prior knowledge with new evidence in a way predicted by Bayesian statistics.

Bayesian methods also apply to decision-making in cognitive contexts (16). What would be your guess of the life expectancy of a 39-year-old? People can use two sources of information to answer this question. They may use the prior, the distribution of lifetimes (Fig. 2B, green). They may also use the likelihood, this person must have survived the first 39 years of his or her life (Fig. 2B, red). With Bayes's rule, we can combine these two pieces of information, just as in the example of tennis (Fig. 2B, black). We thus estimate a life expectancy of about 76 years (using the negative square error as the utility function). Human participants exhibit cognitive behaviors that are close to optimal predictions. The same approach has been used to successfully predict human estimates for many other everyday cognition problems (16). People incorporate knowledge about the probability distributions into estimates in a fashion that is predicted by Bayesian decision theory.

Cue combination. Estimation will often depend on two different cues. For example, we may see and feel an object and use both senses to infer the properties of the object. Bayesian statistics allows us to solve these problems with the same mathematical framework used for the combination of prior and likelihood. A couple of recent studies have examined how subjects solve such cue combination problems. For example, the combinations of visual and auditory information and visual and tactile information, as well as within modality cue combination (e.g., texture and disparity), have been studied. In such cases, cues are combined in a fashion that is close to the optimum prescribed by Bayesian statistics (14, 15, 17, 18).

Although there is strong evidence that animals represent their degree of uncertainty and combine different pieces of information into a joint estimate, there is still relatively little known about how the nervous system implements such computations. Recently, a couple of theoretical studies proposed how the nervous system may represent probabilities and combine them into estimates (19, 20). What is exciting about these studies is that they make clear testable predictions of how the nervous system may represent uncertainty. Understanding this representation is key for neuroscience because animals essentially make all their decisions in the presence of uncertainty.

Integration of information over time. The state of the world and our information about it is continually changing. In the example of tennis, we only used an estimate at a single point of time. In reality, tennis players continuously observe the ball's motion, updating their beliefs about the position where the ball will land in a continuous manner. The Kalman filter formalizes how such a process may work. The filter uses knowledge about the dynamics of the world to convert its belief about the state of the world at the previous instance of time into a belief at a future point in time. For example, in the tennis game we expect the ball to move because of its forward momentum and to accelerate because of gravity. The resulting belief (prior, Fig. 3, green) is then combined with new sensory information (likelihood, Fig. 3, red) to produce an updated belief (posterior, Fig. 3, ellipses). This way the Kalman filter updates its beliefs over time. In the case of playing tennis, such a strategy predicts that, as the ball flies, our estimate of the landing position will progressively be updated and usually become more precise. It is thus possible to make an efficient decision in situations that change continuously over time.

A range of recent studies have probed the strategies used by human participants in such situations. The way that people estimate the position of their hand in the dark is well predicted by Kalman filtering (21). Similarly, when people balance a pole they also seem to use such strategies (22). Moreover, the way the nervous system represents changing muscle properties can be understood by assuming a Kalman filtering strategy (23, 24). These studies demonstrate that when people integrate information over time to make simple sensorimotor decisions, they seem to do so in a fashion that is consistent with the optimal Bayesian solution.

The nervous system constantly integrates information over time, and a range of new studies analyzed how it does so (25). In many such experiments, one of two stimuli is given, for example either a stimulus that moves to the right or a stimulus that moves to the left. If the stimulus is sufficiently noisy, the nervous system needs to integrate information over an extended period of time to make a good decision. Neurons were found that exhibit activities that correlate with the predicted process of optimal information integration over time. The nervous system takes into account probabilistic knowledge of potential rewards when integrating evidence for decision-making (26). The resulting models are particularly useful because they have a normative component (optimal integration of evidence) while having a straightforward descriptive component (neurons can integrate inputs over time).

## Inverse Decision Theory

If people make decisions optimally, the mechanism of decision theory may be inverted: Computational techniques [e.g., (27)] are used to infer which priors, likelihoods, and utility functions the participants used to make their decisions. For example, a utility function with a few free parameters may be proposed, and the parameters may be fit to human behavior. Experimental economics (28) has extensively asked which utility functions people are optimizing. Only recently has the study of neuroscience and low-level decision-making started asking which priors are used and which utility functions are optimized.

Many studies in neuroscience analyze motor control, an area where decision-making is key. For example, would you rather carry a 2-kg weight for 1 min or a 1-kg weight for 2 min? We intuitively make repeatable choices in such situations that are of relevance to everyday life. Such a utility function was recently inferred by using inverse decision theory (29) (Fig. 4A). This utility function is highly nonlinear in force magnitude and duration and is more complicated than previously proposed utility functions (30). This highlights a problem in decision theory: Frequently, good fits to behavior may be obtained with wrong utility functions. Inverse decision theory can thus be seen as a way of searching for violations of the assumptions made when building a decision theoretic model.

The framework of inverse decision theory also allows the analysis of which priors and which likelihoods are used by people. Studies indicate that people underestimate the speed of visual motion (31). This has been argued as the result of using a Gaussian prior for interpreting that low speeds are most likely. But why should people use a Gaussian distribution? We do not know which prior would be optimal in real life, although some recent progress in the statistics of natural scenes may lead that way (32). In a recent experiment, the prior used by human participants was measured by using inverse decision theory (33) (Fig. 4A). The prior is not Gaussian and is rather similar to an exponential function. This may inspire future experiments to characterize how the nervous system implements such a prior.

The strength of inverse decision theory in allowing for a wide range of possible utility functions is also its weakness. Inverse decision theory will always yield a utility function, likelihood, or prior for which the actually observed behavior is optimal. If the results differ from those assumed by a previous decision theoretic model, we can falsify this model. However, the results should only be the basis of a new model if we can understand (and test) how the inferred functions derive from properties of the world. Similar problems also appear in other decision theoretic models that do not explicitly use inverse decision theory. The theoretician may fiddle with the decision theoretic model, trying different utilities, likelihoods, and priors, until there are good fits to human performance. As in all other models that explain data, overfitting is also a problem for decision theoretic models.

Inverse decision theory allows for the estimations of the used utility functions, priors, and likelihoods, which may alleviate the search for their neural representation. For example, if we know how utility functions depend on ideas of fairness when playing games, it is possible to search for brain areas that represent this aspect of the utility function (11). Searching for the neural representation of a utility function that has been proposed on theoretical grounds, but is irrelevant for human behavior, may miss important aspects of decision-making. When searching for the representation of priors or utility functions in the nervous system, it seems central to know the form of the priors and utility functions that are actually used by human participants.

## Discussion

The world is complicated, and consequently so is deciding. Models in neuroscience typically analyze simple relationships between variables. However, at least our high-level behavior is characterized by structural relationships. Events in the world have causes, and we naturally interpret events in terms of cause and effect. A few studies of Bayesian statistics over the past couple of years have started to address the issue of how people may be able to infer the structure of the world (34, 35). Structure implies that not only features but their relationships play a fundamental role. This concept has long been at the heart of cognitive science. How people solve complicated real-world problems needs to be understood. A lot of recent progress in machine learning aims at inferring the structure of the world from real data (36), a process that people perform effortlessly. The study of decisions in neuroscience can draw upon advances in machine-learning to make interesting new predictions.

The decision theoretic approach may be limited in several ways. Humans should behave suboptimally for ethologically new kinds of decisions that are not repeated enough to allow for learning [but see (37)]. This may, for example, be relevant to the way people participate in lotteries (38). Using Bayes's rule to combine pieces of information is the best mathematical solution to any information combination problem and thus always has been the best solution. Similarly, moving efficiently has always been beneficial to animals. The simple low-level properties of neural decision-making should thus be expected to be close to optimal.

Decision theory formalizes how animals should decide and thus does not directly make predictions of how the nervous system should implement the algorithm leading to such decisions (39). Countless different implementations may lead to the same optimal decision rules. However, the normative approach is not limited to decision theory. There may equally be costs and benefits to implementing algorithms in various ways. For example, having many long connections between neurons may use volume and energy and lead to slow information transmission. The idea of wiring length minimization explains well the placement of neurons in cortical maps (40) as well as within the body of the nematode Caenorhabditis elegans (41). Similarly, it may be argued that visual neurons should faithfully represent the world with the fewest spikes (least metabolic cost), an idea that predicts many properties of sensory representations (42). Normative ideas may even apply to cellular properties. The density of sodium channels in the squid giant axon may be understood from the idea that the squid giant axon should rapidly transmit action potentials used to allow the squid to flee from a predator (43, 44). Normative models and decision theory in particular offer ways of formalizing important problems that the nervous system needs to solve. Models in neuroscience should seek to explain the wealth of available experimental data and also incorporate knowledge of the problem solved by the system.

View Abstract