title: Evaluating hippocampal internal architecture on MRI: inter-rater reliability of a proposed scoring system.
abstract: BACKGROUND
  Asymmetry of hippocampal internal architecture (HIA) has been reported to be a frequent imaging finding in epilepsy patients with temporal lobe epilepsy (TLE) who exhibit other signs of hippocampal sclerosis. HIA asymmetry may also be an independent predictor of the side of seizure onset in patients with otherwise normal MRI scans. The study of HIA asymmetry and its relationship to the laterality of TLE would benefit from a reliable method of assessing the clarity of HIA in MRI scans. We propose a visual scoring system that rates HIA clarity from 1 (imperceptible) to 4 (excellent) and report the inter-rater reliability (IRR) of this system.
  
  
  METHODS
  In the initial preliminary phase of this study we examined IRR using a kappa statistic (κ) among a mixed group of expert and non-expert reviewers using only a brief description of the scoring system to score single images from a series of patients. In the second phase we explored the effect of training on the use of our HIA scoring system by assessing IRR among neuroimaging experts before and after a brief interactive training session. In this phase, multiple slices from each patient were scored. Separate κ values and intraclass correlation coefficients (ICC) were calculated from the scores given to each hippocampal image and from the asymmetry of scores between left and right for each slice. In the third phase the effect of training on non-expert reviewers was explored using a similar approach as with the expert reviewers.
  
  
  RESULTS
  In the preliminary phase of the study, HIA scoring of single images showed substantial agreement among expert reviewers (κHIA=0.65), fair agreement among non-expert reviewers (κHIA=0.27), and a fair to moderate degree of agreement among all the reviewers as a whole (κHIA=0.40). In the second phase, prior to training there was substantial agreement among expert reviewers in regard to the individual HIA scores (κHIA=0.62; ICCHIA=0.81) but only moderate agreement on the degree of asymmetry (κAsym=0.47; ICCAsym=0.71). Training improved agreement on the individual HIA scores (κHIA=0.58-0.72; ICCHIA=0.76-0.84) and on the degree of asymmetry (κAsym=0.61-0.67; ICCAsym=0.81-0.85). Among non-expert reviewers, scores improved from only a fair degree of agreement pre-training (κHIA=0.25, κAsym=0.25; ICCHIA=0.68, ICCAsym=0.66) to a moderate level of agreement after training (κHIA=0.54, κAsym=0.52; ICCHIA=0.78, ICCAsym=0.81).
  
  
  CONCLUSIONS
  The proposed HIA scoring system has a substantial degree of inter-rater reliability among experienced neuroimaging reviewers. Training improves the detection of asymmetries in HIA score in particular. Non-expert reviewers can employ the system with a moderate degree of reliability, and training has an even greater impact on the improvement of scoring reliability.
authors.affiliation: UAB Epilepsy Center, Department of Neurology, University of Alabama at Birmingham, Birmingham, AL, United States. LVERHOEF@UAB.EDU
authors.forename: Lawrence W
authors.initials: LW
authors.lastname: Ver Hoef
authors.forename: A LeBron
authors.initials: AL
authors.lastname: Paige
authors.forename: Kristen O
authors.initials: KO
authors.lastname: Riley
authors.forename: Joel
authors.initials: J
authors.lastname: Cure
authors.forename: Mehdi
authors.initials: M
authors.lastname: Soltani
authors.forename: Frank B
authors.initials: FB
authors.lastname: Williams
authors.forename: Richard E
authors.initials: RE
authors.lastname: Kennedy
authors.forename: Jerzy P
authors.initials: JP
authors.lastname: Szaflarski
authors.forename: Robert C
authors.initials: RC
authors.lastname: Knowlton
mesh_terms: D000293:Adolescent; D000328:Adult; D017668:Age of Onset; D016477:Artifacts; D003627:Data Interpretation, Statistical; D004833:Epilepsy, Temporal Lobe / Q000473:pathology*; D005260:Female; D006624:Hippocampus / Q000473:pathology*; D006801:Humans; D007090:Image Interpretation, Computer-Assisted; D008279:Magnetic Resonance Imaging / Q000379:methods*; D008297:Male; D008875:Middle Aged; D015588:Observer Variation; D015203:Reproducibility of Results; D055815:Young Adult
keywords: Ammon's horn; HIA; HS; Hippocampus; IRR; Inter-rater reliability; Internal architecture; MRI; TLE; Temporal lobe epilepsy; hippocampal internal architecture; hippocampal sclerosis; inter-rater reliability; temporal lobe epilepsy
references.citation: AJNR Am J Neuroradiol. 1993 May-Jun;14(3):753-62
references.pmid: 8517369
references.citation: AJNR Am J Neuroradiol. 2010 Oct;31(9):1682-9
references.pmid: 20538822
references.citation: Neurology. 1998 Nov;51(5):1256-62
references.pmid: 9818842
references.citation: Neurology. 1998 Aug;51(2):465-71
references.pmid: 9710020
references.citation: Biometrics. 2002 Mar;58(1):209-15
references.pmid: 11890316
references.citation: Neuroimage. 2009 Aug 1;47(1):42-9
references.pmid: 19303448
references.citation: Phys Ther. 2005 Mar;85(3):257-68
references.pmid: 15733050
references.citation: Neurology. 1994 Jan;44(1):42-6
references.pmid: 8290088
references.citation: Brain. 1966 Sep;89(3):499-530
references.pmid: 5922048
references.citation: Biometrics. 1977 Mar;33(1):159-74
references.pmid: 843571
references.citation: Epilepsia. 2009 Jun;50(6):1474-83
references.pmid: 19400880
references.citation: J Magn Reson Imaging. 2008 Nov;28(5):1266-72
references.pmid: 18972336
references.citation: Epilepsy Res. 2013 Sep;106(1-2):155-63
references.pmid: 23911210
references.citation: Psychol Bull. 1979 Mar;86(2):420-8
references.pmid: 18839484