title: Outcome measures based on classification performance fail to predict the intelligibility of binary-masked speech.
abstract: To date, the most commonly used outcome measure for assessing ideal binary mask estimation algorithms is based on the difference between the hit rate and the false alarm rate (H-FA). Recently, the error distribution has been shown to substantially affect intelligibility. However, H-FA treats each mask unit independently and does not take into account how errors are distributed. Alternatively, algorithms can be evaluated with the short-time objective intelligibility (STOI) metric using the reconstructed speech. This study investigates the ability of H-FA and STOI to predict intelligibility for binary-masked speech using masks with different error distributions. The results demonstrate the inability of H-FA to predict the behavioral intelligibility and also illustrate the limitations of STOI. Since every estimation algorithm will make errors that are distributed in different ways, performance evaluations should not be made solely on the basis of these metrics.
authors.affiliation: Hearing Systems Group, Department of Electrical Engineering, Technical University of Denmark, DK-2800 Kongens Lyngby, Denmark.
authors.forename: Abigail Anne
authors.initials: AA
authors.lastname: Kressner
authors.affiliation: Hearing Systems Group, Department of Electrical Engineering, Technical University of Denmark, DK-2800 Kongens Lyngby, Denmark.
authors.forename: Tobias
authors.initials: T
authors.lastname: May
authors.affiliation: School of Electrical and Computer Engineering, 777 Atlantic Drive NW, Georgia Institute of Technology, Atlanta, Georgia 30332, USA.
authors.forename: Christopher J
authors.initials: CJ
authors.lastname: Rozell
mesh_terms: D000161:Acoustic Stimulation; D000465:Algorithms; D001302:Audiometry, Speech; D003054:Cochlear Implants; D004558:Electric Stimulation; D006801:Humans; D009622:Noise / Q000009:adverse effects*; D010470:Perceptual Masking; D021641:Recognition, Psychology; D012815:Signal Processing, Computer-Assisted; D013061:Speech Acoustics; D013065:Speech Intelligibility; D014833:Voice Quality