title: Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography.
abstract: OBJECTIVE
  To assess the diagnostic performance of distributed human intelligence for the classification of polyp candidates identified with computer-aided detection (CAD) for computed tomographic (CT) colonography.
  
  
  METHODS
  This study was approved by the institutional Office of Human Subjects Research. The requirement for informed consent was waived for this HIPAA-compliant study. CT images from 24 patients, each with at least one polyp of 6 mm or larger, were analyzed by using CAD software to identify 268 polyp candidates. Twenty knowledge workers (KWs) from a crowdsourcing platform labeled each polyp candidate as a true or false polyp. Two trials involving 228 KWs were conducted to assess reproducibility. Performance was assessed by comparing the area under the receiver operating characteristic curve (AUC) of KWs with the AUC of CAD for polyp classification.
  
  
  RESULTS
  The detection-level AUC for KWs was 0.845 ± 0.045 (standard error) in trial 1 and 0.855 ± 0.044 in trial 2. These were not significantly different from the AUC for CAD, which was 0.859 ± 0.043. When polyp candidates were stratified by difficulty, KWs performed better than CAD on easy detections; AUCs were 0.951 ± 0.032 in trial 1, 0.966 ± 0.027 in trial 2, and 0.877 ± 0.048 for CAD (P = .039 for trial 2). KWs who participated in both trials showed a significant improvement in performance going from trial 1 to trial 2; AUCs were 0.759 ± 0.052 in trial 1 and 0.839 ± 0.046 in trial 2 (P = .041).
  
  
  CONCLUSIONS
  The performance of distributed human intelligence is not significantly different from that of CAD for colonic polyp classification.
authors.affiliation: Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, Radiology and Imaging Sciences, National Institutes of Health Clinical Center, Bethesda, MD 20892-1182, USA.
authors.forename: Tan B
authors.initials: TB
authors.lastname: Nguyen
authors.forename: Shijun
authors.initials: S
authors.lastname: Wang
authors.forename: Vishal
authors.initials: V
authors.lastname: Anugu
authors.forename: Natalie
authors.initials: N
authors.lastname: Rose
authors.forename: Matthew
authors.initials: M
authors.lastname: McKenna
authors.forename: Nicholas
authors.initials: N
authors.lastname: Petrick
authors.forename: Joseph E
authors.initials: JE
authors.lastname: Burns
authors.forename: Ronald M
authors.initials: RM
authors.lastname: Summers
mesh_terms: D000368:Aged; D000465:Algorithms; D019540:Area Under Curve; D003111:Colonic Polyps / Q000000981:diagnostic imaging*; D023881:Colonography, Computed Tomographic / Q000379:methods*; D005260:Female; D006801:Humans; D021621:Imaging, Three-Dimensional; D020407:Internet; D008297:Male; D008875:Middle Aged; D012372:ROC Curve; D011857:Radiographic Image Interpretation, Computer-Assisted / Q000379:methods*; D015203:Reproducibility of Results; D012189:Retrospective Studies; D012984:Software; D018709:Statistics, Nonparametric