title: Is the Juice Worth the Squeeze? Costs and Benefits of Multiple Human Annotators for Clinical Text De-identification.
abstract: BACKGROUND
  Clinical text contains valuable information but must be de-identified before it can be used for secondary purposes. Accurate annotation of personally identifiable information (PII) is essential to the development of automated de-identification systems and to manual redaction of PII. Yet the accuracy of annotations may vary considerably across individual annotators and annotation is costly. As such, the marginal benefit of incorporating additional annotators has not been well characterized.
  
  
  OBJECTIVE
  This study models the costs and benefits of incorporating increasing numbers of independent human annotators to identify the instances of PII in a corpus. We used a corpus with gold standard annotations to evaluate the performance of teams of annotators of increasing size.
  
  
  METHODS
  Four annotators independently identified PII in a 100-document corpus consisting of randomly selected clinical notes from Family Practice clinics in a large integrated health care system. These annotations were pooled and validated to generate a gold standard corpus for evaluation.
  
  
  RESULTS
  Recall rates for all PII types ranged from 0.90 to 0.98 for individual annotators to 0.998 to 1.0 for teams of three, when meas-ured against the gold standard. Median cost per PII instance discovered during corpus annotation ranged from $ 0.71 for an individual annotator to $ 377 for annotations discovered only by a fourth annotator.
  
  
  CONCLUSIONS
  Incorporating a second annotator into a PII annotation process reduces unredacted PII and improves the quality of annotations to 0.99 recall, yielding clear benefit at reasonable cost; the cost advantages of annotation teams larger than two diminish rapidly.
authors.affiliation: David S. Carrell, PhD, Group Health Research Institute, 1730 Minor Ave, Suite 1600, Seattle, WA 98101, USA, E-mail: carrell.d@ghc.org.
authors.forename: David S
authors.initials: DS
authors.lastname: Carrell
authors.forename: David J
authors.initials: DJ
authors.lastname: Cronkite
authors.forename: Bradley A
authors.initials: BA
authors.lastname: Malin
authors.forename: John S
authors.initials: JS
authors.lastname: Aberdeen
authors.forename: Lynette
authors.initials: L
authors.lastname: Hirschman
mesh_terms: D003362:Cost-Benefit Analysis / Q000191:economics*; D057225:Data Mining / Q000191:economics*; D057286:Electronic Health Records; D006801:Humans; D010355:Patient Identification Systems / Q000191:economics*
keywords: Patient data privacy; cost analysis; data sharing; natural language processing