title: Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods.
abstract: Sleep spindles are discrete, intermittent patterns of brain activity observed in human electroencephalographic data. Increasingly, these oscillations are of biological and clinical interest because of their role in development, learning and neurological disorders. We used an Internet interface to crowdsource spindle identification by human experts and non-experts, and we compared their performance with that of automated detection algorithms in data from middle- to older-aged subjects from the general population. We also refined methods for forming group consensus and evaluating the performance of event detectors in physiological data such as electroencephalographic recordings from polysomnography. Compared to the expert group consensus gold standard, the highest performance was by individual experts and the non-expert group consensus, followed by automated spindle detectors. This analysis showed that crowdsourcing the scoring of sleep data is an efficient method to collect large data sets, even for difficult tasks such as spindle identification. Further refinements to spindle detection algorithms are needed for middle- to older-aged subjects.
authors.affiliation: Center for Sleep Science and Medicine, Stanford University, Stanford, California, USA.
authors.forename: Simon C
authors.initials: SC
authors.lastname: Warby
authors.affiliation: 1] Center for Sleep Science and Medicine, Stanford University, Stanford, California, USA. [2] Danish Center for Sleep Medicine, Glostrup University Hospital, Glostrup, Denmark.
authors.forename: Sabrina L
authors.initials: SL
authors.lastname: Wendt
authors.affiliation: Computational Vision Laboratory, California Institute of Technology, Pasadena, California, USA.
authors.forename: Peter
authors.initials: P
authors.lastname: Welinder
authors.affiliation: 1] Center for Sleep Science and Medicine, Stanford University, Stanford, California, USA. [2] Danish Center for Sleep Medicine, Glostrup University Hospital, Glostrup, Denmark.
authors.forename: Emil G S
authors.initials: EG
authors.lastname: Munk
authors.affiliation: Center for Sleep Science and Medicine, Stanford University, Stanford, California, USA.
authors.forename: Oscar
authors.initials: O
authors.lastname: Carrillo
authors.affiliation: Department of Electrical Engineering, Technical University of Denmark, Kongens Lyngby, Denmark.
authors.forename: Helge B D
authors.initials: HB
authors.lastname: Sorensen
authors.affiliation: Danish Center for Sleep Medicine, Glostrup University Hospital, Glostrup, Denmark.
authors.forename: Poul
authors.initials: P
authors.lastname: Jennum
authors.affiliation: Department of Population Health Sciences, University of Wisconsin-Madison, Madison, Wisconsin, USA.
authors.forename: Paul E
authors.initials: PE
authors.lastname: Peppard
authors.affiliation: Computational Vision Laboratory, California Institute of Technology, Pasadena, California, USA.
authors.forename: Pietro
authors.initials: P
authors.lastname: Perona
authors.affiliation: Center for Sleep Science and Medicine, Stanford University, Stanford, California, USA.
authors.forename: Emmanuel
authors.initials: E
authors.lastname: Mignot
mesh_terms: D000368:Aged; D000465:Algorithms; D001331:Automation; D063045:Crowdsourcing; D004569:Electroencephalography; D006801:Humans; D020407:Internet; D008875:Middle Aged; D012894:Sleep Stages / Q000502:physiology*