title: Intuitive visualizations of pitch and loudness in speech.
abstract: Visualizing acoustic features of speech has proven helpful in speech therapy; however, it is as yet unclear how to create intuitive and fitting visualizations. To better understand the mappings from speech sound aspects to visual space, a large web-based experiment (n = 249) was performed to evaluate spatial parameters that may optimally represent pitch and loudness of speech. To this end, five novel animated visualizations were developed and presented in pairwise comparisons, together with a static visualization. Pitch and loudness of speech were each mapped onto either the vertical (y-axis) or the size (z-axis) dimension, or combined (with size indicating loudness and vertical position indicating pitch height) and visualized as an animation along the horizontal dimension (x-axis) over time. The results indicated that firstly, there is a general preference towards the use of the y-axis for both pitch and loudness, with pitch ranking higher than loudness in terms of fit. Secondly, the data suggest that representing both pitch and loudness combined in a single visualization is preferred over visualization in only one dimension. Finally, the z-axis, although not preferred, was evaluated as corresponding better to loudness than to pitch. This relation between sound and visual space has not been reported previously for speech sounds, and elaborates earlier findings on musical material. In addition to elucidating more general mappings between auditory and visual modalities, the findings provide us with a method of visualizing speech that may be helpful in clinical applications such as computerized speech therapy, or other feedback-based learning paradigms.
authors.affiliation: Health, Medical and Neuropsychology Unit, Institute of Psychology, Leiden University, Leiden, The Netherlands. r.s.schaefer@fsw.leidenuniv.nl.|Leiden Institute for Brain and Cognition (LIBC), Leiden University, Leiden, The Netherlands. r.s.schaefer@fsw.leidenuniv.nl.
authors.forename: Rebecca S
authors.initials: RS
authors.lastname: Schaefer
authors.affiliation: Sint Maartenskliniek Research, Nijmegen, The Netherlands.
authors.forename: Lilian J
authors.initials: LJ
authors.lastname: Beijer
authors.affiliation: Sint Maartenskliniek Research, Nijmegen, The Netherlands.|Donders Institute for Brain, Cognition and Behaviour, Centre for Cognition, Nijmegen, The Netherlands.
authors.forename: Wiel
authors.initials: W
authors.lastname: Seuskens
authors.affiliation: Sint Maartenskliniek Research, Nijmegen, The Netherlands.|Department of Linguistics, Radboud University Nijmegen, Nijmegen, The Netherlands.
authors.forename: Toni C M
authors.initials: TC
authors.lastname: Rietveld
authors.affiliation: Donders Institute for Brain, Cognition and Behaviour, Centre for Cognition, Nijmegen, The Netherlands.|The Institute for Logic, Language and Computation, University of Amsterdam, Amsterdam, The Netherlands.
authors.forename: Makiko
authors.initials: M
authors.lastname: Sadakata
mesh_terms: D000328:Adult; D000368:Aged; D005260:Female; D006801:Humans; D008144:Loudness Perception / Q000502:physiology*; D008297:Male; D008875:Middle Aged; D010898:Pitch Perception / Q000502:physiology*; D013067:Speech Perception / Q000502:physiology*; D014796:Visual Perception / Q000502:physiology*; D055815:Young Adult
keywords: Audio-visual processing; Feedback learning; Speech therapy; Visualizing sound