The neural response to emotional prosody, as revealed by functional magnetic resonance imaging

Abstract

Prosody is an important feature of language, comprising intonation, loudness, and tempo. Emotional prosodic processing forms an integral part of our social interactions. The main aim of this study was to use bold contrast fMRI to clarify the normal functional neuroanatomy of emotional prosody, in passive and active contexts. Subjects performed six separate scanning studies, within which two different conditions were contrasted: (1) “pure” emotional prosody versus rest; (2) congruent emotional prosody versus ‘neutral’ sentences; (3) congruent emotional prosody versus rest; (4) incongruent emotional prosody versus rest; (5) congruent versus incongruent emotional prosody; and (6) an active experiment in which subjects were instructed to either attend to the emotion conveyed by semantic content or that conveyed by tone of voice. Data resulting from these contrasts were analysed using SPM99. Passive listening to emotional prosody consistently activated the lateral temporal lobe (superior and/or middle temporal gyri). This temporal lobe response was relatively right-lateralised with or without semantic information. Both the separate and direct comparisons of congruent and incongruent emotional prosody revealed that subjects used fewer brain regions to process incongruent emotional prosody than congruent. The neural response to attention to semantics, was left lateralised, and recruited an extensive network not activated by attention to emotional prosody. Attention to emotional prosody modulated the response to speech, and induced right-lateralised activity, including the middle temporal gyrus. In confirming the results of lesion and neuropsychological studies, the current study emphasises the importance of the right hemisphere in the processing of emotional prosody, specifically the lateral temporal lobes.

Introduction

When listening to speech, we rely upon a range of cues upon which to base our inference as to the communicative intent of others. To interpret the meaning of speech, how something is said may be as important as what is actually said. Features of speech such as intonation, loudness, and tempo are collectively termed prosody, which was first defined in a clinical context by Monrad-Kohn (1947) as “that faculty of speech which conveys different shades of meaning by means of variations in stress and pitch—irrespective of the words and grammatical construction”. Many writers make a distinction between linguistic functions of prosody (e.g. to differentiate between a statement and a question) and non-linguistic (e.g. emotion and attitude), but distinctions between linguistic and non-linguistic functions of prosody are equivocal in terms of linguistic form (Ladd, 1996). Whilst linguistic functions are usually conveyed by categorical pitch direction, non-linguistic functions, such as the indication of a speaker’s emotional intent, are usually conveyed by gradient features, e.g. pitch height and range, and rhythmicity. However, there is an overlap, in that pitch direction can contribute to non-linguistic functions as well as linguistic (see the writings of Ladd (1996) for an in depth discussion of these issues).

Historical accounts of the neural basis of the response to emotion as signalled by prosody, can be traced back to the writings of Hughlings-Jackson (1931), who described patients in whom emotional speech was preserved despite aphasia resulting from lesions in the left hemisphere. These findings led to the suggestion that emotional prosody is subserved by the right hemisphere, a supposition that has received further support from studies of lesion-induced language deficits (Pell, 1999; Schmitt et al., 1997, Starkstein et al., 1994). Compared to normal controls, patients with right hemisphere lesions speak with a more monotone voice (Pell, 1999). Such patients have great difficulty expressing emotional intention through modulation of their tone of voice. Furthermore, patients who find it difficult to discriminate and identify emotional prosody exhibit a significantly higher frequency of right hemisphere lesions than left hemisphere lesions (Starkstein et al., 1994). Unlike patients suffering from right hemisphere lesions, patients suffering from left hemisphere lesions demonstrate a near normal ability to recognise the emotion expressed by prosody relative to normal controls (Schmitt et al., 1997).

Although the majority of neurological studies suggest right hemisphere mediation of the response to emotional prosody, a number of studies have failed to distinguish clearly between the left and right hemispheres’ involvement in emotional prosodic processing. It has been claimed that emotional prosody is not invariably disrupted by right hemisphere damage (Lebrun, Lessines, De Vresse & Leleux, 1985), and that it may also be affected by left hemisphere damage (Seron et al., 1982, Speedie et al., 1984). Van Lancker & Sidtis (1992) suggest that hemispheric differences in the ability to interpret emotional prosody are qualitative rather than quantitative. They argue that prosodic processes are made up of multiple skills and functions distributed across both hemispheres. According to this model, the left and right hemispheres are responsible for processing different aspects of emotional prosody; whereas the right hemisphere is superior than the left at extracting pitch information, the left is superior to the right at extracting temporal information. Thus, the pattern of lateralisation observed may, to some extent, depend on the manner in which emotional prosodic processing is assessed, i.e. on the components of emotional prosody emphasised by the task demands. Since lexical semantic information (hereafter referred to as ‘semantics’) is primarily left hemisphere mediated (Vikingstad, George, Johnson & Cao, 2000), it may also be the case that inconsistent laterality findings relate to the degree to which different groups of subjects use semantic information to judge the emotion conveyed. Although some studies have used emotional prosody stimuli filtered to remove semantic information, previous studies have not examined the influence of competing or incongruent semantic information. A study incorporating stimuli in which the emotions conveyed by semantic and prosodic information were incongruent, could establish whether semantic information can affect laterality of response to emotional prosody even when the task required subjects to ignore semantic information, and focus on the emotion conveyed by prosody. This question could also be answered by actively examining the effects on lateralisation of response to speech containing emotional prosody, of switching attention between semantic and prosodic aspects of the stimuli. Further inconsistencies in the literature derive from the fact that in the patients studied, the precise location of right hemisphere damage has varied, and thus it has not always been clear which structures in the right hemisphere are critical for the mediation of emotional prosody.

A popular hypothesis is that the organisation of prosodic functions in the right hemisphere mirrors that of language functions in the left (Gorelick and Ross, 1987, Ross and Mesulam, 1979; Ross, 1981). The two frontoparietal lesion patients examined by Ross & Mesulam (1979) both had difficulty in portraying a specific emotion through modulation of vocal intonation. In contrast, their ability to identify the emotion portrayed in normal speech was unaffected. Ross et al.’s model thus proposes that anterior regions (e.g. frontal lobes) mediate the expression of emotional prosody, but that its comprehension is affected by posterior (temporal lobe) lesions. Whilst dichotic listening studies of healthy controls have also provided evidence for a right-lateralised response to emotional prosody (Herrero and Hillix, 1990, Ley and Bryden, 1982; Schmitt et al., 1997) dichotic listening tells us little about the underlying pattern of regional activity.

Many functional imaging studies have examined the neuroanatomical basis of non-prosodic language, and much is known about the brain’s response to phonemic (Ravizza, 2001), semantic (Raymer et al., 1997), syntactic (Sakai, Hashimoto & Homae, 2001) and propositional aspects of speech (Posner & Pavese, 1998). In contrast, comparatively few studies have used functional imaging techniques to clarify the brain’s response to emotional prosody. When people make judgements about the emotion conveyed by emotional prosody, they appear to primarily rely on fundamental frequency or pitch (Scherer, 1986). Consistent with right hemisphere mechanisms being involved in the perception of emotional prosody, Zatorre, Evans, Meyer & Gjedde (1992) used PET to demonstrate right prefrontal cortex activation (inferior and middle frontal gyri) when the response to speech was subtracted from the response to pitch. Again using PET, George et al. (1996) sought to determine whether regions responding to emotional prosody comprehension differed from those involved in understanding emotion based on semantic content. Understanding semantic content activated the middle frontal gyrus bilaterally, but on the left more than on the right. In contrast, responding to the emotional prosody activated the inferior frontal gyrus, but only in the right hemisphere. Taking advantage of fMRI’s superior spatial resolution, Buchanan et al. (2000) confirmed that emotional prosody is normally lateralised to frontal and temporal right hemisphere regions, using a task that required subjects to discriminate words based on emotional intonation or phonemic characteristics. It is clear from these functional imaging studies, that several right hemisphere regions are recruited when comprehending emotional prosody, particularly in the lateral frontal and temporal cortices.

The aforementioned imaging studies suggest that the brain regions involved in processing emotional prosody may be wider ranging than those suggested by lesion studies; the role of non-temporal lobe regions needs particular clarification. In addition, little is currently known about how the mediation of emotional prosody is influenced by other cognitive processes such as the processing of conflicting emotional information or the accuracy of processing. The aims of the current study were (i) to increase our understanding of the relative contributions of the regional and hemispheric activations involved in the cortical response to emotional prosody, and (ii) to determine what can be learnt about the functional neuroanatomy of emotional prosodic processing when emotional contours are dissociated from their lexical items and attentive processes.

According to Ross and coworkers model (Gorelick and Ross, 1987, Ross and Mesulam, 1979) the organization of affective language in the right hemisphere mirrors that of propositional language in the left hemisphere. It is proposed that the right middle temporal gyrus will perform similar integrative and interpretative functions when processing emotional prosody to that which occurs during the processing of semantic and non-prosodic auditory stimuli by the left middle temporal gyrus. It is unlikely that emotional prosody is processed solely by secondary auditory association cortices, such as the superior temporal gyrus. The emotional qualities of emotional prosody are likely to require processing by non-auditory regions. Integration of the emotional, auditory and other information contained in emotional prosody may necessitate the advanced associative properties of the middle temporal gyrus. Indeed, the middle temporal gyrus integrates information from various sources (including past memories, emotional and auditory information), allowing complex and wide-ranging decisions about the meaning and interpretation of speech (Ojemann & Schoenfield-McNeill, 1998; Perani et al., 1999, Raymer et al., 1997).

It was therefore hypothesised that, within the right hemisphere, emotional prosody would primarily activate the middle temporal gyrus.