Characteristics of EEG Interpreters Associated With Higher Interrater Agreement.
OBJECTIVE
The goal of the project is to determine characteristics of academic neurophysiologist EEG interpreters (EEGers), which predict good interrater agreement (IRA) and to determine the number of EEGers needed to develop an ideal standardized testing and training data set for epileptiform transient (ET) detection algorithms.


METHODS
A three-phase scoring method was used. In phase 1, 19 EEGers marked the location of ETs in two hundred 30-second segments of EEG from 200 different patients. In phase 2, EEG events marked by at least 2 EEGers were annotated by 18 EEGers on a 5-point scale to indicate whether they were ETs. In phase 3, a third opinion was obtained from EEGers on any inconsistencies between phase 1 and phase 2 scoring.


RESULTS
The IRA for the 18 EEGers was only fair. A select group of the EEGers had good IRA and the other EEGers had low IRA. Board certification by the American Board of Clinical Neurophysiology was associated with better IRA performance but other board certifications, years of fellowship training, and years of practice were not. As the number of EEGers used for scoring is increased, the amount of change in the consensus opinion decreases steadily and is quite low as the group size approaches 10.


CONCLUSIONS
The IRA among EEGers varies considerably. The EEGers must be tested before use as scorers for ET annotation research projects. The American Board of Clinical Neurophysiology certification is associated with improved performance. The optimal size for a group of experts scoring ETs in EEG is probably in the 6 to 10 range.
*Department of Neurology, Medical University of South Carolina, Charleston, South Carolina, U.S.A.; †Department of Neurology, Vanderbilt University, Nashville, Tennessee, U.S.A.; ‡Department of Neurology, University of Texas Health Science Center of Huston, Huston, Texas, U.S.A.; §Mission Health, Asheville, North Carolina, U.S.A.; ‖Department of Neurology, Wayne State University and Detroit Medical Center, Detroit, Michigan, U.S.A.; ¶Academic Health System, Hamad Medical Corporation, Doha, Qatar; #Norton Neurology Services, Louisville, Kentucky, U.S.A.; **Department of Public Health Sciences, Medical University of South Carolina, Charleston, South Carolina, U.S.A.; and ††School of Computing, Clemson University, Clemson, South Carolina, U.S.A.
Jonathan J
JJ
Halford
Amir
A
Arain
Giridhar P
GP
Kalamangalam
Suzette M
SM
LaRoche
Bonilha
B
Leonardo
Maysaa
M
Basha
Nabil J
NJ
Azar
Ekrem
E
Kutluay
Gabriel U
GU
Martz
Wolf J
WJ
Bethany
Chad G
CG
Waters
Brian C
BC
Dean
D000465:Algorithms; D001921:Brain / Q000503:physiopathology; D004569:Electroencephalography / Q000379:methods*; D004827:Epilepsy / Q000175:diagnosis* / Q000503:physiopathology; D006801:Humans; D015588:Observer Variation; D012815:Signal Processing, Computer-Assisted; D012984:Software
Epilepsy Behav. 2007 Nov;11(3):257-62
17719853
Clin Neurophysiol. 2016 Jan;127(1):102-107
25802204
Neurology. 2002 Feb 12;58(3):493-5
11839862
J Neurosci Methods. 2013 Jan 30;212(2):308-16
23174094
J Clin Neurophysiol. 2010 Feb;27(1):12-6
20087204
Electroencephalogr Clin Neurophysiol. 1996 Mar;98(3):186-98
8631278
Arch Neurol. 1992 Mar;49(3):231-7
1536624
J Clin Neurophysiol. 2011 Apr;28(2):178-84
21399515
Comput Biol Med. 2008 Jul;38(7):805-16
18550047
Electroencephalogr Clin Neurophysiol. 1992 May;82(5):391-3
1374708
Med Biol Eng Comput. 2008 Mar;46(3):263-72
18071771
Clin Neurophysiol. 2009 Nov;120(11):1909-15
19836303
Epilepsia. 2006;47 Suppl 1:23-5
17044821
Br J Math Stat Psychol. 2008 May;61(Pt 1):29-48
18482474
Clin Electroencephalogr. 2000 Jul;31(3):122-30
10923198