title: A unified framework for evaluating the risk of re-identification of text de-identification tools.
abstract: It has become regular practice to de-identify unstructured medical text for use in research using automatic methods, the goal of which is to remove patient identifying information to minimize re-identification risk. The metrics commonly used to determine if these systems are performing well do not accurately reflect the risk of a patient being re-identified. We therefore developed a framework for measuring the risk of re-identification associated with textual data releases.
  
  
  
  We apply the proposed evaluation framework to a data set from the University of Michigan Medical School. Our risk assessment results are then compared with those that would be obtained using a typical contemporary micro-average evaluation of recall in order to illustrate the difference between the proposed evaluation framework and the current baseline method.
  
  
  
  We demonstrate how this framework compares against common measures of the re-identification risk associated with an automated text de-identification process. For the probability of re-identification using our evaluation framework we obtained a mean value for direct identifiers of 0.0074 and a mean value for quasi-identifiers of 0.0022. The 95% confidence interval for these estimates were below the relevant thresholds. The threshold for direct identifier risk was based on previously used approaches in the literature. The threshold for quasi-identifiers was determined based on the context of the data release following commonly used de-identification criteria for structured data.
  
  
  
  Our framework attempts to correct for poorly distributed evaluation corpora, accounts for the data release context, and avoids the often optimistic assumptions that are made using the more traditional evaluation approach. It therefore provides a more realistic estimate of the true probability of re-identification.
  
  
  
  This framework should be used as a basis for computing re-identification risk in order to more realistically evaluate future text de-identification tools.
authors.affiliation: School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada; Privacy Analytics Inc., Ottawa, Canada.
authors.forename: Martin
authors.initials: M
authors.lastname: Scaiano
authors.affiliation: Privacy Analytics Inc., Ottawa, Canada.
authors.forename: Grant
authors.initials: G
authors.lastname: Middleton
authors.affiliation: Children's Hospital of Eastern Ontario Research Institute, Ottawa, Canada.
authors.forename: Luk
authors.initials: L
authors.lastname: Arbuckle
authors.affiliation: School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada; Privacy Analytics Inc., Ottawa, Canada.
authors.forename: Varada
authors.initials: V
authors.lastname: Kolhatkar
authors.affiliation: School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada.
authors.forename: Liam
authors.initials: L
authors.lastname: Peyton
authors.affiliation: Michigan Institute for Data Science (MIDAS), University of Michigan Medical School, Office of Research, Ann Arbor, United States.
authors.forename: Moira
authors.initials: M
authors.lastname: Dowling
authors.affiliation: Department of Pediatrics, University of Michigan, Ann Arbor, United States.
authors.forename: Debbie S
authors.initials: DS
authors.lastname: Gipson
authors.affiliation: School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada; Privacy Analytics Inc., Ottawa, Canada; Department of Pediatrics, University of Ottawa, Ottawa, Canada; Children's Hospital of Eastern Ontario Research Institute, Ottawa, Canada. Electronic address: kelemam@uottawa.ca.
authors.forename: Khaled
authors.initials: K
authors.lastname: El Emam
mesh_terms: D003219:Confidentiality; D000068240:Data Anonymization; D057286:Electronic Health Records; D006801:Humans; D012306:Risk
keywords: Data sharing; De-identification; Evaluation framework; Medical text; Natural language processing; Re-identification risk