title: Preparing an annotated gold standard corpus to share with extramural investigators for de-identification research.
abstract: OBJECTIVE
  The current study aims to fill the gap in available healthcare de-identification resources by creating a new sharable dataset with realistic Protected Health Information (PHI) without reducing the value of the data for de-identification research. By releasing the annotated gold standard corpus with Data Use Agreement we would like to encourage other Computational Linguists to experiment with our data and develop new machine learning models for de-identification. This paper describes: (1) the modifications required by the Institutional Review Board before sharing the de-identification gold standard corpus; (2) our efforts to keep the PHI as realistic as possible; (3) and the tests to show the effectiveness of these efforts in preserving the value of the modified data set for machine learning model development.
  
  
  METHODS
  In a previous study we built an original de-identification gold standard corpus annotated with true Protected Health Information (PHI) from 3503 randomly selected clinical notes for the 22 most frequent clinical note types of our institution. In the current study we modified the original gold standard corpus to make it suitable for external sharing by replacing HIPAA-specified PHI with newly generated realistic PHI. Finally, we evaluated the research value of this new dataset by comparing the performance of an existing published in-house de-identification system, when trained on the new de-identification gold standard corpus, with the performance of the same system, when trained on the original corpus. We assessed the potential benefits of using the new de-identification gold standard corpus to identify PHI in the i2b2 and PhysioNet datasets that were released by other groups for de-identification research. We also measured the effectiveness of the i2b2 and PhysioNet de-identification gold standard corpora in identifying PHI in our original clinical notes.
  
  
  RESULTS
  Performance of the de-identification system using the new gold standard corpus as a training set was very close to training on the original corpus (92.56 vs. 93.48 overall F-measures). Best i2b2/PhysioNet/CCHMC cross-training performances were obtained when training on the new shared CCHMC gold standard corpus, although performances were still lower than corpus-specific trainings.
  
  
  CONCLUSIONS
  We successfully modified a de-identification dataset for external sharing while preserving the de-identification research value of the modified gold standard corpus with limited drop in machine learning de-identification performance.
authors.affiliation: Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA.
authors.forename: Louise
authors.initials: L
authors.lastname: Deleger
authors.affiliation: Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA.
authors.forename: Todd
authors.initials: T
authors.lastname: Lingren
authors.affiliation: Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA.
authors.forename: Yizhao
authors.initials: Y
authors.lastname: Ni
authors.affiliation: Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA.
authors.forename: Megan
authors.initials: M
authors.lastname: Kaiser
authors.affiliation: Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA.
authors.forename: Laura
authors.initials: L
authors.lastname: Stoutenborough
authors.affiliation: Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA.
authors.forename: Keith
authors.initials: K
authors.lastname: Marsolo
authors.affiliation: Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA.
authors.forename: Michal
authors.initials: M
authors.lastname: Kouril
authors.affiliation: Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA.
authors.forename: Katalin
authors.initials: K
authors.lastname: Molnar
authors.affiliation: Division of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA.
authors.forename: Imre
authors.initials: I
authors.lastname: Solti
mesh_terms: D016494:Computer Security; D057286:Electronic Health Records; D020408:Health Insurance Portability and Accountability Act; D008490:Medical Informatics; D014481:United States
keywords: Automated de-identification; De-identification gold standard; Health insurance portability and accountability act; Natural Language Processing; Privacy of patient data; Protected Health Information