Communication Goes Multimodal

See allHide authors and affiliations

Science  26 Feb 1999:
Vol. 283, Issue 5406, pp. 1272-1273
DOI: 10.1126/science.283.5406.1272

The signals that organisms exchange as they communicate are often very complex. Understanding how these signals are perceived poses special problems both for physiologists who study neural integration and for behavioral scientists interested in communication. This “binding problem”—how an organism creates a coherent percept from parts of a stimulus analyzed separately—is especially acute when several sensory modalities are used. Communication researchers tend to categorize signals by the primary sensory channel involved, but in reality multiple channels are often engaged simultaneously, especially in highly social, group-living creatures. Despite predictions that multiple, concurrent stimuli should be important (1), their influence on signal efficacy and meaning has only recently been fully appreciated (2, 3). For example, human speech perception is influenced by visual stimuli (4), and signals as diverse as threat expressions of macaques (see upper figure) and recruitment signals of ants have different consequences, depending on the combination of sensory modalities used (5, 6).

Visual and vocal.

Bimodal threat display of an adult male rhesus macaque (Macaca mulatta) combining a facial expression (open-mouth threat) with a vocalization (bark). Image was digitized from videotape. Insert depicts the bark (y axis, frequency 0 to 8 kHz; × axis, time 0 to 2 s) taken from videotape at the same moment as the picture.

Behavioral neuroscientists find that integration of information from multiple sensory channels is crucial for attention and perception in humans, monkeys, birds, and insects, particularly in the processing of stimuli associated with posture and movement (7). The communicative consequences of combining signal components from different sensory channels remain poorly understood, and we lack a theoretical framework for dealing with them. Here we offer a classification system for categorizing and comparing the effects of multimodal signals (see lower figure).

Classification of multimodal signals.

Redundant signals are depicted above, nonredundant signals below. (Left) Responses to two separate components (a and b) represented by geometric shapes (the same shape indicates the same qualitative response; different shapes indicate different responses). (Right) Responses to the combined multimodal signal.

The components of a multimodal signal may be either redundant or nonredundant in meaning. Redundancy is common (8) and ensures that the message will get through in the face of environmental noise (backup signals) (9). Nonredundant components have the advantage of providing more information per unit time (multiple messages) (10). Both types can be distinguished empirically by the behavior they elicit from a recipient. When presented separately, redundant signal components should have equivalent effects on a receiver, whereas nonredundant components should have different effects (5).

When components are combined simultaneously into a multimodal signal, several outcomes are possible (see lower figure). Redundant components might result in the same response as each component alone. Courting male moths (Cycnia tenera) elicit equivalent responses from females regardless of whether their pheromones and ultrasonic sounds are presented separately or together (11). More commonly, the combination of redundant components results in an enhanced response. Aphaenogaster ants recruit help for carrying prey by emitting pheromones, but with large prey they also stridulate, producing a substrate-borne vibrational signal. The stridulation has a small effect alone, but both components together recruit more workers (6). Similar multiplicative effects occur during neural processing of simultaneous visual and auditory stimuli in the superior colliculus of cats (12).

Combinations of nonredundant components yield other outcomes. The two components could be independent, eliciting distinct responses even when combined. Pheromones from female Cupiennius salei spiders alert males to the presence of a potential mate. Concomitant vibrational signals from the female then direct males to her location (13). These two components function independently whether they are perceived simultaneously or not. Alternatively, one component may dominate the other. Dogs signal play behavior visually with a bow, and sometimes also growl, normally a threat. Separately, these signals are contradictory, but their combination elicits play, the visual component taking precedence (14).

One nonredundant component may modulate the effect of the other. Male Alpheus heterochaeli shrimp respond aggressively to visual cues alone, such as an open claw, but do not respond to chemical cues alone. When the two are combined and the pheromone is from a female, male aggressive responses are suppressed (15). Neural analogs for modulation include cells in the cat superior colliculus that respond to visual stimuli alone but not to auditory stimuli alone. Auditory and visual stimuli together elicit enhanced responses in some neurons, leave some unchanged, and leave others depressed (12).

Finally, the combination of two nonredundant components can produce an entirely new response (emergence). When a vocal stimulus (human phoneme “ba”) is mismatched with a visual stimulus (face articulating “ga”), subjects may perceive a new phoneme, “da” (4). Aromatic pyrazines and red and yellow coloration are commonly associated with noxious insects. Presented alone, neither cue produces aversion in chicks; aversion appears only when the odor and color occur simultaneously (2). Here, multimodal stimuli evoke a response not elicited by the unimodal components. Similarly, some cat superior colliculus cells respond to multimodal but not unimodal stimuli (12). The provision of a common terminology for discussion of multimodal signaling and its underlying integrative neural processing may encourage efforts to unify physiologic and behavioral research in this area.

References and Notes

View Abstract

Stay Connected to Science

Navigate This Article