title: Effects of personal identifier resynthesis on clinical text de-identification.
abstract: OBJECTIVE
  De-identified medical records are critical to biomedical research. Text de-identification software exists, including "resynthesis" components that replace real identifiers with synthetic identifiers. The goal of this research is to evaluate the effectiveness and examine possible bias introduced by resynthesis on de-identification software.
  
  
  METHODS
  We evaluated the open-source MITRE Identification Scrubber Toolkit, which includes a resynthesis capability, with clinical text from Vanderbilt University Medical Center patient records. We investigated four record classes from over 500 patients' files, including laboratory reports, medication orders, discharge summaries and clinical notes. We trained and tested the de-identification tool on real and resynthesized records.
  
  
  METHODS
  We measured performance in terms of precision, recall, F-measure and accuracy for the detection of protected health identifiers as designated by the HIPAA Safe Harbor Rule.
  
  
  RESULTS
  The de-identification tool was trained and tested on a collection of real and resynthesized Vanderbilt records. Results for training and testing on the real records were 0.990 accuracy and 0.960 F-measure. The results improved when trained and tested on resynthesized records with 0.998 accuracy and 0.980 F-measure but deteriorated moderately when trained on real records and tested on resynthesized records with 0.989 accuracy 0.862 F-measure. Moreover, the results declined significantly when trained on resynthesized records and tested on real records with 0.942 accuracy and 0.728 F-measure.
  
  
  CONCLUSIONS
  The de-identification tool achieves high accuracy when training and test sets are homogeneous (ie, both real or resynthesized records). The resynthesis component regularizes the data to make them less "realistic," resulting in loss of performance particularly when training on resynthesized data and testing on real data.
authors.affiliation: Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Turkey.
authors.forename: Reyyan
authors.initials: R
authors.lastname: Yeniterzi
authors.forename: John
authors.initials: J
authors.lastname: Aberdeen
authors.forename: Samuel
authors.initials: S
authors.lastname: Bayer
authors.forename: Ben
authors.initials: B
authors.lastname: Wellner
authors.forename: Lynette
authors.initials: L
authors.lastname: Hirschman
authors.forename: Bradley
authors.initials: B
authors.lastname: Malin
mesh_terms: D001185:Artificial Intelligence; D016494:Computer Security; D003219:Confidentiality; D057286:Electronic Health Records; D006801:Humans; D016247:Information Storage and Retrieval; D012984:Software; D014481:United States