title: Bootstrapping a de-identification system for narrative patient records: cost-performance tradeoffs.
abstract: OBJECTIVE
  We describe an experiment to build a de-identification system for clinical records using the open source MITRE Identification Scrubber Toolkit (MIST). We quantify the human annotation effort needed to produce a system that de-identifies at high accuracy.
  
  
  METHODS
  Using two types of clinical records (history and physical notes, and social work notes), we iteratively built statistical de-identification models by annotating 10 notes, training a model, applying the model to another 10 notes, correcting the model's output, and training from the resulting larger set of annotated notes. This was repeated for 20 rounds of 10 notes each, and then an additional 6 rounds of 20 notes each, and a final round of 40 notes. At each stage, we measured precision, recall, and F-score, and compared these to the amount of annotation time needed to complete the round.
  
  
  RESULTS
  After the initial 10-note round (33min of annotation time) we achieved an F-score of 0.89. After just over 8h of annotation time (round 21) we achieved an F-score of 0.95. Number of annotation actions needed, as well as time needed, decreased in later rounds as model performance improved. Accuracy on history and physical notes exceeded that of social work notes, suggesting that the wider variety and contexts for protected health information (PHI) in social work notes is more difficult to model.
  
  
  CONCLUSIONS
  It is possible, with modest effort, to build a functioning de-identification system de novo using the MIST framework. The resulting system achieved performance comparable to other high-performing de-identification systems.
authors: Department of Pediatrics, University of Michigan, Ann Arbor, MI, USA. hanauer@med.umich.edu
authors: David
authors: D
authors: Hanauer
authors: John
authors: J
authors: Aberdeen
authors: Samuel
authors: S
authors: Bayer
authors: Benjamin
authors: B
authors: Wellner
authors: Cheryl
authors: C
authors: Clark
authors: Kai
authors: K
authors: Zheng
authors: Lynette
authors: L
authors: Hirschman
mesh_terms: D016494:Computer Security; D003219:Confidentiality; D057286:Electronic Health Records / Q000191:economics* / Q000592:standards*; D006801:Humans; D033181:Information Dissemination; D012984:Software
keywords: Computerized [E05.318.308.940.968.625]; Electronic health records [E05.318.308.940.968.625.500]; Medical informatics [L01.313.500]; Medical record systems; NLP; Natural language processing [L01.224.065.580]; Privacy [I01.880.604.473.352.500]