The neural response to emotional prosody, as revealed by functional magnetic resonance imaging
Introduction
When listening to speech, we rely upon a range of cues upon which to base our inference as to the communicative intent of others. To interpret the meaning of speech, how something is said may be as important as what is actually said. Features of speech such as intonation, loudness, and tempo are collectively termed prosody, which was first defined in a clinical context by Monrad-Kohn (1947) as “that faculty of speech which conveys different shades of meaning by means of variations in stress and pitch—irrespective of the words and grammatical construction”. Many writers make a distinction between linguistic functions of prosody (e.g. to differentiate between a statement and a question) and non-linguistic (e.g. emotion and attitude), but distinctions between linguistic and non-linguistic functions of prosody are equivocal in terms of linguistic form (Ladd, 1996). Whilst linguistic functions are usually conveyed by categorical pitch direction, non-linguistic functions, such as the indication of a speaker’s emotional intent, are usually conveyed by gradient features, e.g. pitch height and range, and rhythmicity. However, there is an overlap, in that pitch direction can contribute to non-linguistic functions as well as linguistic (see the writings of Ladd (1996) for an in depth discussion of these issues).
Historical accounts of the neural basis of the response to emotion as signalled by prosody, can be traced back to the writings of Hughlings-Jackson (1931), who described patients in whom emotional speech was preserved despite aphasia resulting from lesions in the left hemisphere. These findings led to the suggestion that emotional prosody is subserved by the right hemisphere, a supposition that has received further support from studies of lesion-induced language deficits (Pell, 1999; Schmitt et al., 1997, Starkstein et al., 1994). Compared to normal controls, patients with right hemisphere lesions speak with a more monotone voice (Pell, 1999). Such patients have great difficulty expressing emotional intention through modulation of their tone of voice. Furthermore, patients who find it difficult to discriminate and identify emotional prosody exhibit a significantly higher frequency of right hemisphere lesions than left hemisphere lesions (Starkstein et al., 1994). Unlike patients suffering from right hemisphere lesions, patients suffering from left hemisphere lesions demonstrate a near normal ability to recognise the emotion expressed by prosody relative to normal controls (Schmitt et al., 1997).
Although the majority of neurological studies suggest right hemisphere mediation of the response to emotional prosody, a number of studies have failed to distinguish clearly between the left and right hemispheres’ involvement in emotional prosodic processing. It has been claimed that emotional prosody is not invariably disrupted by right hemisphere damage (Lebrun, Lessines, De Vresse & Leleux, 1985), and that it may also be affected by left hemisphere damage (Seron et al., 1982, Speedie et al., 1984). Van Lancker & Sidtis (1992) suggest that hemispheric differences in the ability to interpret emotional prosody are qualitative rather than quantitative. They argue that prosodic processes are made up of multiple skills and functions distributed across both hemispheres. According to this model, the left and right hemispheres are responsible for processing different aspects of emotional prosody; whereas the right hemisphere is superior than the left at extracting pitch information, the left is superior to the right at extracting temporal information. Thus, the pattern of lateralisation observed may, to some extent, depend on the manner in which emotional prosodic processing is assessed, i.e. on the components of emotional prosody emphasised by the task demands. Since lexical semantic information (hereafter referred to as ‘semantics’) is primarily left hemisphere mediated (Vikingstad, George, Johnson & Cao, 2000), it may also be the case that inconsistent laterality findings relate to the degree to which different groups of subjects use semantic information to judge the emotion conveyed. Although some studies have used emotional prosody stimuli filtered to remove semantic information, previous studies have not examined the influence of competing or incongruent semantic information. A study incorporating stimuli in which the emotions conveyed by semantic and prosodic information were incongruent, could establish whether semantic information can affect laterality of response to emotional prosody even when the task required subjects to ignore semantic information, and focus on the emotion conveyed by prosody. This question could also be answered by actively examining the effects on lateralisation of response to speech containing emotional prosody, of switching attention between semantic and prosodic aspects of the stimuli. Further inconsistencies in the literature derive from the fact that in the patients studied, the precise location of right hemisphere damage has varied, and thus it has not always been clear which structures in the right hemisphere are critical for the mediation of emotional prosody.
A popular hypothesis is that the organisation of prosodic functions in the right hemisphere mirrors that of language functions in the left (Gorelick and Ross, 1987, Ross and Mesulam, 1979; Ross, 1981). The two frontoparietal lesion patients examined by Ross & Mesulam (1979) both had difficulty in portraying a specific emotion through modulation of vocal intonation. In contrast, their ability to identify the emotion portrayed in normal speech was unaffected. Ross et al.’s model thus proposes that anterior regions (e.g. frontal lobes) mediate the expression of emotional prosody, but that its comprehension is affected by posterior (temporal lobe) lesions. Whilst dichotic listening studies of healthy controls have also provided evidence for a right-lateralised response to emotional prosody (Herrero and Hillix, 1990, Ley and Bryden, 1982; Schmitt et al., 1997) dichotic listening tells us little about the underlying pattern of regional activity.
Many functional imaging studies have examined the neuroanatomical basis of non-prosodic language, and much is known about the brain’s response to phonemic (Ravizza, 2001), semantic (Raymer et al., 1997), syntactic (Sakai, Hashimoto & Homae, 2001) and propositional aspects of speech (Posner & Pavese, 1998). In contrast, comparatively few studies have used functional imaging techniques to clarify the brain’s response to emotional prosody. When people make judgements about the emotion conveyed by emotional prosody, they appear to primarily rely on fundamental frequency or pitch (Scherer, 1986). Consistent with right hemisphere mechanisms being involved in the perception of emotional prosody, Zatorre, Evans, Meyer & Gjedde (1992) used PET to demonstrate right prefrontal cortex activation (inferior and middle frontal gyri) when the response to speech was subtracted from the response to pitch. Again using PET, George et al. (1996) sought to determine whether regions responding to emotional prosody comprehension differed from those involved in understanding emotion based on semantic content. Understanding semantic content activated the middle frontal gyrus bilaterally, but on the left more than on the right. In contrast, responding to the emotional prosody activated the inferior frontal gyrus, but only in the right hemisphere. Taking advantage of fMRI’s superior spatial resolution, Buchanan et al. (2000) confirmed that emotional prosody is normally lateralised to frontal and temporal right hemisphere regions, using a task that required subjects to discriminate words based on emotional intonation or phonemic characteristics. It is clear from these functional imaging studies, that several right hemisphere regions are recruited when comprehending emotional prosody, particularly in the lateral frontal and temporal cortices.
The aforementioned imaging studies suggest that the brain regions involved in processing emotional prosody may be wider ranging than those suggested by lesion studies; the role of non-temporal lobe regions needs particular clarification. In addition, little is currently known about how the mediation of emotional prosody is influenced by other cognitive processes such as the processing of conflicting emotional information or the accuracy of processing. The aims of the current study were (i) to increase our understanding of the relative contributions of the regional and hemispheric activations involved in the cortical response to emotional prosody, and (ii) to determine what can be learnt about the functional neuroanatomy of emotional prosodic processing when emotional contours are dissociated from their lexical items and attentive processes.
According to Ross and coworkers model (Gorelick and Ross, 1987, Ross and Mesulam, 1979) the organization of affective language in the right hemisphere mirrors that of propositional language in the left hemisphere. It is proposed that the right middle temporal gyrus will perform similar integrative and interpretative functions when processing emotional prosody to that which occurs during the processing of semantic and non-prosodic auditory stimuli by the left middle temporal gyrus. It is unlikely that emotional prosody is processed solely by secondary auditory association cortices, such as the superior temporal gyrus. The emotional qualities of emotional prosody are likely to require processing by non-auditory regions. Integration of the emotional, auditory and other information contained in emotional prosody may necessitate the advanced associative properties of the middle temporal gyrus. Indeed, the middle temporal gyrus integrates information from various sources (including past memories, emotional and auditory information), allowing complex and wide-ranging decisions about the meaning and interpretation of speech (Ojemann & Schoenfield-McNeill, 1998; Perani et al., 1999, Raymer et al., 1997).
It was therefore hypothesised that, within the right hemisphere, emotional prosody would primarily activate the middle temporal gyrus.
Section snippets
Subjects
Thirteen healthy controls were recruited from the staff and students of the University of Manchester. Subjects were aged 18–60 (mean 32.2±0.93), did not have a history of hearing difficulties, and spoke English as their first language. In view of the fact that gender and handedness may both influence language lateralisation (Beaton, 1997), only male and right-handed subjects were studied, since they are the most clearly lateralised for language, and since recruiting half men and half women
Results
Results are summarised in Table 2, Table 3.
Summary of results
The right-lateral temporal lobe (middle and/or superior temporal gyri) was activated in response to pure emotional prosody versus rest, congruent emotional prosody versus neutral, congruent emotional prosody versus rest, and incongruent emotional prosody versus rest. The left lateral temporal lobe was also activated in response to pure, congruent and incongruent emotional prosody versus rest. Overall, in the emotional prosody conditions, temporal lobe activation was relatively
Conclusions
In confirming the results of lesion and neuropsychological studies, the current study emphasises the importance of the right hemisphere in the processing of emotional prosody, specifically the middle and superior temporal gyri. The healthy brain appears to invoke neural systems to process speech containing emotional intonation distinct from those that process neutral and non-emotional speech.
Acknowledgments
This study was performed whilst Rachel Mitchell was receiving a Ph.D. studentship from Neuraxis. We would also like to thank Professor Deakin at the Neuroscience and Psychiatry Unit, University of Manchester for generous departmental funding.
References (62)
The relation of planum temporale asymmetry and morphology of the corpus callosum to handedness, gender, and dyslexia: a review of the evidence
Brain and Language
(1997)The perception of stress and lateralization of prosody
Brain and Language
(1985)- et al.
Auditory event-related potentials, dichotic listening performance and handedness as indices of lateralisation in dyslexic and normal readers
International Journal of Psychophysiology
(1994) - et al.
Recognition of emotional prosody and verbal components of spoken language: an fMRI study
Brain Research Cognitive Brain Research
(2000) - et al.
Procedural learning in schizophrenia: a functional magnetic resonance imaging investigation
Schizophrenia Research
(2002) - et al.
Neuroanatomic overlap of working memory and spatial attention networks: a functional MRI comparison within subjects
Neuroimage
(1999) - et al.
Dysprosody and the non-dominant hemisphere
Language Sciences
(1985) - et al.
A dissociation of right and left hemispheric effects for recognizing emotional tone and verbal content
Brain and Cognition
(1982) - et al.
Detection power, estimation efficiency, and predictability in event-related fMRI
Neuroimage
(2001) - et al.
Hemispheric asymmetry in lexical decisions: the effects of grammatical class and imageability
Brain and Language
(1999)
Neurons in human temporal cortex active with verbal associative learning
Brain and Language
The temporal organization of affective and non-affective speech in patients with right-hemisphere infarcts
Cortex
Word and picture matching: a PET study of semantic category effects
Neuropsychologia
Relating selective brain damage to impairments with voicing contrasts
Brain and Language
Cognitive neuropsychological analysis and neuroanatomic correlates in a case of anomia
Brain and Language
Sentence processing in the cerebral cortex
Neuroscience Research
Hemispheric asymmetry in the recognition of emotional attitude conveyed by facial expression, prosody and propositional speech
Cortex
Decoding paralinguistic signals: effect of semantic and prosodic cues on aphasics’ comprehension
Journal of Communication Disorders
Hemispheric differences in grammatical class
Brain and Language
Cortical language lateralisation in right handed normal subjects using functional magnetic resonance imaging
Journal of the Neurological Sciences
Apathy and depressed mood in acquired brain damage: relationship to lesion localization and psychophysiological reactivity
Psychological Medicine
A classification of hand preference by association analysis
British Journal of Psychology
Function of the left planum temporale in auditory and linguistic processing
Brain
Right hemisphere emotional perception: evidence across multiple channels
Neuropsychology
The role of the right hemisphere in the interpretation of figurative aspects of language: a positron emission tomography activation study
Brain
The nonverbal affect lexicon: theoretical perspectives from neuropsychological studies of affect perception
Neuropsychology
Affective verbal memory in patients with temporal lobe epilepsy
Applied Neuropsychology
Hemispheric asymmetry for emotional stimuli detected with fMRI
NeuroReport
Cortical language activation in stroke patients recovering from aphasia with functional MRI
Stroke
Single-shot T2(∗) measurement to establish optimum echo time for fMRI: studies of the visual, motor, and auditory cortices at 3.0 T
Magnetic Resonance in Medicine
Updating working memory for words: a PET activation study
Human Brain Mapping
Cited by (277)
The emotional component of inner speech: A pilot exploratory fMRI study
2023, Brain and CognitionThe hearing hippocampus
2022, Progress in NeurobiologyAuditory emotion recognition deficits in schizophrenia: A systematic review and meta-analysis
2021, Asian Journal of PsychiatryLesion loci of impaired affective prosody: A systematic review of evidence from stroke
2021, Brain and Cognition