title: Spike detection: Inter-reader agreement and a statistical Turing test on a large data set.
abstract: OBJECTIVE
  Compare the spike detection performance of three skilled humans and three computer algorithms.
  
  
  METHODS
  40 prolonged EEGs, 35 containing reported spikes, were evaluated. Spikes and sharp waves were marked by the humans and algorithms. Pairwise sensitivity and false positive rates were calculated for each human-human and algorithm-human pair. Differences in human pairwise performance were calculated and compared to the range of algorithm versus human performance differences as a type of statistical Turing test.
  
  
  RESULTS
  5474 individual spike events were marked by the humans. Mean, pairwise human sensitivities and false positive rates were 40.0%, 42.1%, and 51.5%, and 0.80, 0.97, and 1.99/min. Only the Persyst 13 (P13) algorithm was comparable to humans - 43.9% and 1.65/min. Evaluation of pairwise differences in sensitivity and false positive rate demonstrated that P13 met statistical noninferiority criteria compared to the humans.
  
  
  CONCLUSIONS
  Humans had only a fair level of agreement in spike marking. The P13 algorithm was statistically noninferior to the humans.
  
  
  CONCLUSIONS
  This was the first time that a spike detection algorithm and humans performed similarly. The performance comparison methodology utilized here is generally applicable to problems in which skilled human performance is the desired standard and no external gold standard exists.
authors.affiliation: Persyst Development Corporation, USA. Electronic address: scheuerml@persyst.com.
authors.forename: Mark L
authors.initials: ML
authors.lastname: Scheuer
authors.affiliation: University of Pittsburgh, Department of Neurology, USA.
authors.forename: Anto
authors.initials: A
authors.lastname: Bagic
authors.affiliation: Persyst Development Corporation, USA.
authors.forename: Scott B
authors.initials: SB
authors.lastname: Wilson
mesh_terms: D000200:Action Potentials / Q000502:physiology*; D000465:Algorithms; D001921:Brain / Q000502:physiology*; D016208:Databases, Factual / Q000592:standards; D004569:Electroencephalography / Q000379:methods* / Q000592:standards; D005260:Female; D006801:Humans; D008297:Male; D012189:Retrospective Studies; D012815:Signal Processing, Computer-Assisted
keywords: Artificial neural network; Automated spike detection; EEG; Epileptiform; Inter-reader agreement; Noninferiority