title: Error Rates in Users of Automatic Face Recognition Software.
abstract: In recent years, wide deployment of automatic face recognition systems has been accompanied by substantial gains in algorithm performance. However, benchmarking tests designed to evaluate these systems do not account for the errors of human operators, who are often an integral part of face recognition solutions in forensic and security settings. This causes a mismatch between evaluation tests and operational accuracy. We address this by measuring user performance in a face recognition system used to screen passport applications for identity fraud. Experiment 1 measured target detection accuracy in algorithm-generated 'candidate lists' selected from a large database of passport images. Accuracy was notably poorer than in previous studies of unfamiliar face matching: participants made over 50% errors for adult target faces, and over 60% when matching images of children. Experiment 2 then compared performance of student participants to trained passport officers-who use the system in their daily work-and found equivalent performance in these groups. Encouragingly, a group of highly trained and experienced "facial examiners" outperformed these groups by 20 percentage points. We conclude that human performance curtails accuracy of face recognition systems-potentially reducing benchmark estimates by 50% in operational settings. Mere practise does not attenuate these limits, but superior performance of trained examiners suggests that recruitment and selection of human operators, in combination with effective training and mentorship, can improve the operational accuracy of face recognition systems.
authors.affiliation: School of Psychology, The University of New South Wales, Sydney, Australia.
authors.forename: David
authors.initials: D
authors.lastname: White
authors.affiliation: School of Psychology, The University of New South Wales, Sydney, Australia.
authors.forename: James D
authors.initials: JD
authors.lastname: Dunn
authors.affiliation: School of Psychology, The University of Sydney, Sydney, Australia.
authors.forename: Alexandra C
authors.initials: AC
authors.lastname: Schmid
authors.affiliation: School of Psychology, The University of New South Wales, Sydney, Australia.
authors.forename: Richard I
authors.initials: RI
authors.lastname: Kemp
mesh_terms: D000293:Adolescent; D000328:Adult; D000465:Algorithms; D016208:Databases, Factual; D005006:Ethnicity; D005145:Face; D005260:Female; D006801:Humans; D007091:Image Processing, Computer-Assisted; D008297:Male; D008875:Middle Aged; D012984:Software; D012985:Software Design