Objective: Unconscious fast integration of face and voice information is a crucial brain function necessary for communicating effectively with others. Here, we investigated for evidence of rapid face-voice integration in the auditory cortex. Methods: Magnetic fields (P50m and N100m) evoked by visual stimuli (V), auditory stimuli (A) and audiovisual stimuli (VA), i.e. by face, vowel and simultaneous vowel-face stimuli, were recorded in 22 healthy subjects. Magnetoencephalographic data from 28 channels around bilateral auditory cortices were analyzed. Results: In both hemispheres, AV - V showed significantly larger P50m amplitudes than A. Additionally, compared with A, the N100m amplitudes and dipole moments of AV - V were significantly smaller in the left hemisphere, but not in the right hemisphere. Conclusions: Differential changes in P50m (bilateral) and N100m (left hemisphere) that occur when V (faces) are associated with A (vowel sounds) indicate that AV (face-voice) integration occurs in early processing, likely enabling us to communicate effectively in our lives.

Keywords:
Audiovisual integration, Face, Vowel sound, Magnetoencephalography, N100m, P50m
1.
Belin P, Fecteau S, Bédard C: Thinking the voice: neural correlates of voice perception. Trends Cogn Sci 2004;8:129-135.
2.
Calvert GA, Campbell R, Brammer MJ: Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 2000;10:649-657.
3.
Hertrich I, Dietrich S, Ackermann H: Cross-modal interactions during perception of audiovisual speech and nonspeech signals: an fMRI study. J Cogn Neurosci 2011;23:221-237.
4.
Pekkola J, Ojanen V, Autti T, Jääskeläinen IP, Möttönen R, Sams M: Attention to visual speech gestures enhances hemodynamic activity in the left planum temporale. Hum Brain Mapp 2006;27:471-477.
5.
Van Atteveldt N, Formisano E, Goebel R, Blomert L: Integration of letters and speech sounds in the human brain. Neuron 2004;43:271-282.
6.
Van Wassenhove V, Grant KW, Poeppel D: Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA 2005;102:1181-1186.
7.
McGurk H, MacDonald J: Hearing lips and seeing voices. Nature 1976;264:746-748.
8.
Poeppel D, Phillips C, Yellin E, Rowley HA, Roberts TP, Marantz A: Processing of vowels in supratemporal auditory cortex. Neurosci Lett 1997;221:145-148.
9.
Kirveskari E, Salmelin R, Hari R: Neuromagnetic responses to vowels vs tones reveal hemispheric lateralization. Clin Neurophysiol 2006;117:643-648.
10.
Mäkelä AM, Alku P, Tiitinen H: The auditory N1m reveals the left-hemispheric representation of vowel identity in humans. Neurosci Lett 2003;353:111-114.
11.
Ogata E, Yumoto M, Itoh K, Sekimoto S, Karino S, Kaga K: A magnetoencephalographic study of Japanese vowel processing. Neuroreport 2006;17:1127-1131.
12.
Palva S, Palva JM, Shtyrov Y, Kujala T, Ilmoniemi RJ, Kaila K, Näätänen R: Distinct gamma-band evoked responses to speech and non-speech sounds in humans. J Neurosci 2002;22:RC211.
13.
Hirano S, Hirano Y, Maekawa T, Obayashi C, Oribe N, Kuroki T, Kanba S, Onitsuka T: Abnormal neural oscillatory activity to speech sounds in schizophrenia: a magnetoencephalography study. J Neurosci 2008;28:4897-4903.
14.
Kasai K, Yamada H, Kamio S, Nakagome K, Iwanami A, Fukuda M, Yumoto M, Itoh K, Koshida I: Neuromagnetic correlates of impaired automatic categorical perception of speech sounds in schizophrenia. Schizophr Res 2003;59:159-172.
15.
Oribe N, Onitsuka T, Hirano S, Hirano Y, Maekawa T, Obayashi C, Ueno T, Kasai K, Kanba S: Differentiation between bipolar disorder and schizophrenia revealed by neural oscillation to speech sounds: an MEG study. Bipolar Disord 2010;12:804-812.
16.
De Gelder B, Vroomen J, Annen L, Masthof E, Hodiamont P: Audio-visual integration in schizophrenia. Schizophr Res 2003;59:211-218.
17.
Daskalakis ZJ, Fitzgerald PB, Christensen BK: The role of cortical inhibition in the pathophysiology and treatment of schizophrenia. Brain Res Rev 2007;56:427-442.
18.
Hirano Y, Hirano S, Maekawa T, Obayashi C, Oribe N, Monji A, Kasai K, Kanba S, Onitsuka T: Auditory gating deficit to human voices in schizophrenia: a MEG study. Schizophr Res 2010;117:61-67.
19.
Schulze KK, Hall MH, McDonald C, Marshall N, Walshe M, Murray RM, Bramon E: P50 auditory evoked potential suppression in bipolar disorder patients with psychotic features and their unaffected relatives. Biol Psychiatry 2007;62:121-128.
20.
Grimm S, Escera C: Auditory deviance detection revisited: evidence for a hierarchical novelty system. Int J Psychophysiol 2012;85:88-92.
21.
Talsma D, Doty TJ, Woldorff MG: Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb Cortex 2007;17:679-690.
22.
Lebib R, Papo D, de Bode S, Baudonnière PM: Evidence of a visual-to-auditory cross-modal sensory gating phenomenon as reflected by the human P50 event-related brain potential modulation. Neurosci Lett 2003;341:185-188.
23.
Näätänen R, Picton T: The N1 wave of the human electric and magnetic response to sound: a review and an analysis of the component structure. Psychophysiology 1987;24:375-425.
24.
Besle J, Fort A, Delpuech C, Giard MH: Bimodal speech: early suppressive visual effects in human auditory cortex. Eur J Neurosci 2004;20:2225-2234.
25.
Möttönen R, Schürmann M, Sams M: Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study. Neurosci Lett 2004;363:112-115.
26.
Davis C, Kislyuk D, Kim J, Sams M: The effect of viewing speech on auditory speech processing is different in the left and right hemispheres. Brain Res 2008;1242:151-161.
27.
Arnal LH, Morillon B, Kell CA, Giraud AL: Dual neural routing of visual facilitation in speech processing. J Neurosci 2009;29:13445-13453.
28.
Jääskeläinen IP, Ojanen V, Ahveninen J, Auranen T, Levänen S, Möttönen R, Tarnanen I, Sams M: Adaptation of neuromagnetic N1 responses to phonetic stimuli by visual speech in humans. Neuroreport 2004;15:2741-2744.
29.
Miki K, Watanabe S, Kakigi R: Interaction between auditory and visual stimulus relating to the vowel sounds in the auditory cortex in humans: a magnetoencephalographic study. Neurosci Lett 2004;357:199-202.
30.
Reale RA, Calvert GA, Thesen T, Jenison RL, Kawasaki H, Oya H, Howard MA, Brugge JF: Auditory-visual processing represented in the human superior temporal gyrus. Neuroscience 2007;145:162-184.
31.
Yamamoto T, Williamson SJ, Kaufman L, Nicholson C, Llinás R: Magnetic localization of neuronal activity in the human brain. Proc Natl Acad Sci USA 1988;85:8732-8736.
32.
Oldfield RC: The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 1971;9:97-113.
33.
Hamalainen M, Hari R: Magnetoencephalographic characterization of dynamic brain activation: basic principles and methods of data collection and source analysis; in Toga AW, Mazziotta JC (eds): Brain mapping. The Methods, ed 2. San Diego, Academic Press, 2002, pp 227-254.
34.
Taulu S, Simola J: Spatiotemporal signal space separation method for rejecting nearby interference in MEG measurements. Phys Med Biol 2006;51:1759-1768.
35.
Delorme A, Makeig S: EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 2004;134:9-21.
36.
Tsuchimoto R, Kanba S, Hirano S, Oribe N, Ueno T, Hirano Y, Nakamura I, Oda Y, Miura T, Onitsuka T: Reduced high and low frequency gamma synchronization in patients with chronic schizophrenia. Schizophr Res 2011;133:99-105.
37.
Besle J, Bertrand O, Giard MH: Electrophysiological (EEG, sEEG, MEG) evidence for multiple audiovisual interactions in the human auditory cortex. Hear Res 2009;258:143-151.
38.
Musacchia G, Schroeder CE: Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex. Hear Res 2009;258:72-79.
39.
Jiang W, Stein BE: Cortex controls multisensory depression in superior colliculus. J Neurophysiol 2003;90:2123-2135.
40.
Zikopoulos B, Barbas H: Circuits for multisensory integration and attentional modulation through the prefrontal cortex and the thalamic reticular nucleus in primates. Rev Neurosci 2007;18:417-438.
41.
Möttönen R, Krause CM, Tiippana K, Sams M: Processing of changes in visual speech in the human auditory cortex. Brain Res Cogn Brain Res 2002;13:417-425.
42.
Kayser C, Petkov CI, Logothetis NK: Visual modulation of neurons in auditory cortex. Cereb Cortex 2008;18:1560-1574.
43.
Foxe JJ, Schroeder CE: The case for feedforward multisensory convergence during early cortical processing. Neuroreport 2005;16:419-423.
44.
Schroeder CE, Foxe JJ: The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res Cogn Brain Res 2002;14:187-198.
45.
Szycik GR, Münte TF, Dillo W, Mohammadi B, Samii A, Emrich HM, Dierich DE: Audiovisual integration of speech is disturbed in schizophrenia: an fMRI study. Schizophr Res 2009;110:111-118.
46.
Stekelenburg JJ, Maes JP, Van Gool AR, Sitskoorn M, Vroomen J: Deficient multisensory integration in schizophrenia: an event-related potential study. Schizophr Res 2013;147:253-261.
47.
Calvert GA, Campbell R, Brammer MJ: Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 2000;10:649-657.
48.
Skipper JI, van Wassenhove V, Nusbaum HC, Small SL: Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception. Cereb Cortex 2007;17:2387-2399.
49.
Giard MH, Peronnet F: Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 1999;11:473-490.
50.
Isoğlu-Alkaç U, Kedzior K, Keskindemirci G, Ermutlu N, Karamursel S: Event-related potentials to visual, auditory, and bimodal (combined auditory-visual) stimuli. Int J Neurosci 2007;117:259-273.
© 2015 S. Karger AG, Basel
2015
Copyright / Drug Dosage / Disclaimer
Copyright: All rights reserved. No part of this publication may be translated into other languages, reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, microcopying, or by any information storage and retrieval system, without permission in writing from the publisher.
Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug.
Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.
You do not currently have access to this content.