title: An approach to comparing accuracies of two FLAIR MR sequences in the detection of multiple sclerosis lesions in the brain in the absence of gold standard.
abstract: OBJECTIVE
  The purpose of this study was to present a new methodology to compare accuracies of two imaging fluid attenuated inversion recovery (FLAIR) magnetic resonance sequences in detection of multiple sclerosis (MS) lesions in the brain in the absence of ground truth, and to determine whether the two sequences, which differed only in echo time (TE), have the same accuracy.
  
  
  METHODS
  We acquired FLAIR images at TE(1) = 90 ms and TE(2) = 155 ms from 46 patients with MS (24-69 years old, mean 45.8, 15 males) and 11 healthy volunteers (23-54 years old, mean 37.1, 6 males). Seven experienced neuroradiologists segmented lesions manually on randomly presented corresponding TE(1) and TE(2) images. For every image pair, a "surrogate ground truth" for each TE was generated by applying probability thresholds, ranging from 0.3 to 0.5, to the weighted average of experts' segmentations. Jackknife alternative free-response receiver operating characteristic analysis was used to compare experts' performance on TE(1) and TE(2) images, using successively the TE(1)- and TE(2)-based ground truths.
  
  
  RESULTS
  Supratentorially, there were significant differences in relative accuracy between the two sequences, ranging from 8.4% to 12.1%. In addition, we found a higher ratio of false positives to true positives for the TE(2) sequence using the TE(2) ground truth, compared to the TE(1) equivalent. Infratentorially, differences in the relative accuracy did not reach statistical significance.
  
  
  CONCLUSIONS
  The presented methodology may be useful in assessing the value of new clinical imaging protocols or techniques in the context of replacing existing ones, when the absolute ground truth is not available, and in determining changes in disease progression in follow-up studies. Our results suggest that the sequence with shorter TE should be preferred because it generates relatively fewer false positives. The finding is consistent with results of previous computer simulation studies.
authors.affiliation: Division of Neuroradiology, Department of Radiology, Hospital of the University of Pennsylvania, 3600 Market Street, Philadelphia, PA 19014, USA.
authors.forename: Michel
authors.initials: M
authors.lastname: Bilello
authors.forename: Neeti
authors.initials: N
authors.lastname: Suri
authors.forename: Jaroslaw
authors.initials: J
authors.lastname: Krejza
authors.forename: John H
authors.initials: JH
authors.lastname: Woo
authors.forename: Linda J
authors.initials: LJ
authors.lastname: Bagley
authors.forename: Alexander C
authors.initials: AC
authors.lastname: Mamourian
authors.forename: Arastoo
authors.initials: A
authors.lastname: Vossough
authors.forename: James Y
authors.initials: JY
authors.lastname: Chen
authors.forename: Bradd R
authors.initials: BR
authors.lastname: Millian
authors.forename: Todd
authors.initials: T
authors.lastname: Mulderink
authors.forename: Clyde E
authors.initials: CE
authors.lastname: Markowitz
authors.forename: Elias R
authors.initials: ER
authors.lastname: Melhem
mesh_terms: D000328:Adult; D000368:Aged; D000465:Algorithms; D001921:Brain / Q000473:pathology*; D006801:Humans; D007089:Image Enhancement / Q000379:methods*; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D008279:Magnetic Resonance Imaging / Q000379:methods*; D008297:Male; D009103:Multiple Sclerosis / Q000473:pathology*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity