An approach to comparing accuracies of two FLAIR MR sequences in the detection of multiple sclerosis lesions in the brain in the absence of gold standard.
OBJECTIVE
The purpose of this study was to present a new methodology to compare accuracies of two imaging fluid attenuated inversion recovery (FLAIR) magnetic resonance sequences in detection of multiple sclerosis (MS) lesions in the brain in the absence of ground truth, and to determine whether the two sequences, which differed only in echo time (TE), have the same accuracy.


METHODS
We acquired FLAIR images at TE(1) = 90 ms and TE(2) = 155 ms from 46 patients with MS (24-69 years old, mean 45.8, 15 males) and 11 healthy volunteers (23-54 years old, mean 37.1, 6 males). Seven experienced neuroradiologists segmented lesions manually on randomly presented corresponding TE(1) and TE(2) images. For every image pair, a "surrogate ground truth" for each TE was generated by applying probability thresholds, ranging from 0.3 to 0.5, to the weighted average of experts' segmentations. Jackknife alternative free-response receiver operating characteristic analysis was used to compare experts' performance on TE(1) and TE(2) images, using successively the TE(1)- and TE(2)-based ground truths.


RESULTS
Supratentorially, there were significant differences in relative accuracy between the two sequences, ranging from 8.4% to 12.1%. In addition, we found a higher ratio of false positives to true positives for the TE(2) sequence using the TE(2) ground truth, compared to the TE(1) equivalent. Infratentorially, differences in the relative accuracy did not reach statistical significance.


CONCLUSIONS
The presented methodology may be useful in assessing the value of new clinical imaging protocols or techniques in the context of replacing existing ones, when the absolute ground truth is not available, and in determining changes in disease progression in follow-up studies. Our results suggest that the sequence with shorter TE should be preferred because it generates relatively fewer false positives. The finding is consistent with results of previous computer simulation studies.
Bilello
Michel
M
Division of Neuroradiology, Department of Radiology, Hospital of the University of Pennsylvania, 3600 Market Street, Philadelphia, PA 19014, USA.
Suri
Neeti
N
Krejza
Jaroslaw
J
Woo
John H
JH
Bagley
Linda J
LJ
Mamourian
Alexander C
AC
Vossough
Arastoo
A
Chen
James Y
JY
Millian
Bradd R
BR
Mulderink
Todd
T
Markowitz
Clyde E
CE
Melhem
Elias R
ER
D000328:Adult; D000368:Aged; D000465:Algorithms; D001921:Brain / Q000473:pathology*; D006801:Humans; D007089:Image Enhancement / Q000379:methods*; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D008279:Magnetic Resonance Imaging / Q000379:methods*; D008297:Male; D009103:Multiple Sclerosis / Q000473:pathology*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity