title: Inter-expert and intra-expert reliability in sleep spindle scoring.
abstract: OBJECTIVE
  To measure the inter-expert and intra-expert agreement in sleep spindle scoring, and to quantify how many experts are needed to build a reliable dataset of sleep spindle scorings.
  
  
  METHODS
  The EEG dataset was comprised of 400 randomly selected 115s segments of stage 2 sleep from 110 sleeping subjects in the general population (57±8, range: 42-72 years). To assess expert agreement, a total of 24 Registered Polysomnographic Technologists (RPSGTs) scored spindles in a subset of the EEG dataset at a single electrode location (C3-M2). Intra-expert and inter-expert agreements were calculated as F1-scores, Cohen's kappa (κ), and intra-class correlation coefficient (ICC).
  
  
  RESULTS
  We found an average intra-expert F1-score agreement of 72±7% (κ: 0.66±0.07). The average inter-expert agreement was 61±6% (κ: 0.52±0.07). Amplitude and frequency of discrete spindles were calculated with higher reliability than the estimation of spindle duration. Reliability of sleep spindle scoring can be improved by using qualitative confidence scores, rather than a dichotomous yes/no scoring system.
  
  
  CONCLUSIONS
  We estimate that 2-3 experts are needed to build a spindle scoring dataset with 'substantial' reliability (κ: 0.61-0.8), and 4 or more experts are needed to build a dataset with 'almost perfect' reliability (κ: 0.81-1).
  
  
  CONCLUSIONS
  Spindle scoring is a critical part of sleep staging, and spindles are believed to play an important role in development, aging, and diseases of the nervous system.
authors.affiliation: Center for Sleep Science and Medicine, Stanford University, Palo Alto, CA, United States; Danish Center for Sleep Medicine, Glostrup University Hospital, DK-2600 Glostrup, Denmark.
authors.forename: Sabrina L
authors.initials: SL
authors.lastname: Wendt
authors.affiliation: Computational Vision Laboratory, California Institute of Technology, Pasadena, CA, United States.
authors.forename: Peter
authors.initials: P
authors.lastname: Welinder
authors.affiliation: Dept. of Electrical Engineering, Technical University of Denmark, DK-2800 Kongens Lyngby, Denmark.
authors.forename: Helge B D
authors.initials: HB
authors.lastname: Sorensen
authors.affiliation: Department of Population Health Sciences, University of Wisconsin - Madison, Madison, WI, United States.
authors.forename: Paul E
authors.initials: PE
authors.lastname: Peppard
authors.affiliation: Danish Center for Sleep Medicine, Glostrup University Hospital, DK-2600 Glostrup, Denmark.
authors.forename: Poul
authors.initials: P
authors.lastname: Jennum
authors.affiliation: Computational Vision Laboratory, California Institute of Technology, Pasadena, CA, United States.
authors.forename: Pietro
authors.initials: P
authors.lastname: Perona
authors.affiliation: Center for Sleep Science and Medicine, Stanford University, Palo Alto, CA, United States.
authors.forename: Emmanuel
authors.initials: E
authors.lastname: Mignot
authors.affiliation: Center for Sleep Science and Medicine, Stanford University, Palo Alto, CA, United States; Center for Advanced Research in Sleep Medicine, Hôpital du Sacré-Coeur de Montréal, Department of Psychiatry, Université de Montréal, Montréal, Canada. Electronic address: simon.c.warby@umontreal.ca.
authors.forename: Simon C
authors.initials: SC
authors.lastname: Warby
mesh_terms: D000328:Adult; D000368:Aged; D001143:Arousal / Q000502:physiology*; D004569:Electroencephalography / Q000379:methods*; D005260:Female; D006801:Humans; D008297:Male; D008875:Middle Aged; D015588:Observer Variation; D017286:Polysomnography; D015203:Reproducibility of Results; D012894:Sleep Stages / Q000502:physiology*
keywords: Agreement; Electroencephalography; Event detection; Inter-expert; Inter-rater; Intra-expert; Intra-rater; Polysomnography; Reliability; Sleep scoring; Sleep spindles; Sleep staging