title: Potholes and molehills: bias in the diagnostic performance of diffusion-tensor imaging in concussion.
abstract: OBJECTIVE
  To investigate the extent of bias in a clinical study involving "pothole analysis" of diffusion-tensor imaging (DTI) data used to quantify white matter lesion load in diseases with a heterogeneous spatial distribution of pathologic findings, such as mild traumatic brain injury (TBI), and create a mathematical model of the bias.
  
  
  METHODS
  Use of the same reference population to define normal findings and make comparisons with a patient group introduces bias, which potentially inflates reported diagnostic performance. In this institutional review board-approved prospective observational cohort study, DTI data were obtained in 20 patients admitted to the emergency department with mild TBI and in 16 control subjects. Potholes and molehills were defined as clusters of voxels with fractional anisotropy values more than 2 standard deviations below and above the mean of the corresponding voxels in the reference population, respectively. The number and volume of potholes and molehills in the two groups were compared by using a Mann-Whitney U test.
  
  
  RESULTS
  Standard analysis showed significantly more potholes in mild TBI than in the control group (102.5 ± 34.3 vs 50.6 ± 28.9, P < .001). Repeat analysis by using leave-one-out cross-validation decreased the apparent difference in potholes between groups (mild TBI group, 102.5 ± 34.3; control group, 93.4 ± 27.2; P = .369). It was demonstrated that even with 100 subjects, this bias can decrease the voxelwise false-positive rate by more than 30% in the control group.
  
  
  CONCLUSIONS
  The pothole approach to neuroimaging data may introduce bias, which can be minimized by independent training and test groups or cross-validation methods. This bias is sufficient to call into question the previously reported diagnostic performance of DTI for mild TBI.
authors.affiliation: From the Departments of Radiology (R.W., C.G.F., J.P.N.), Surgery (A.T., K.V.), and Neurology (C.G.F.), University of Vermont, Given Medical Building E301, 89 Beaumont Ave, Burlington, VT 05405.
authors.forename: Richard
authors.initials: R
authors.lastname: Watts
authors.forename: Alex
authors.initials: A
authors.lastname: Thomas
authors.forename: Christopher G
authors.initials: CG
authors.lastname: Filippi
authors.forename: Joshua P
authors.initials: JP
authors.lastname: Nickerson
authors.forename: Kalev
authors.initials: K
authors.lastname: Freeman
mesh_terms: D000293:Adolescent; D000328:Adult; D016880:Anisotropy; D001924:Brain Concussion / Q000175:diagnosis* / Q000473:pathology; D016022:Case-Control Studies; D056324:Diffusion Tensor Imaging / Q000379:methods*; D005260:Female; D015600:Glasgow Coma Scale; D006801:Humans; D007090:Image Interpretation, Computer-Assisted; D008297:Male; D008875:Middle Aged; D009413:Nerve Fibers, Myelinated / Q000473:pathology*; D015588:Observer Variation; D011446:Prospective Studies; D014057:Tomography, X-Ray Computed