title: Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text.
abstract: The Health Insurance Portability and Accountability Act (HIPAA) Safe Harbor method requires removal of 18 types of protected health information (PHI) from clinical documents to be considered "de-identified" prior to use for research purposes. Human review of PHI elements from a large corpus of clinical documents can be tedious and error-prone. Indeed, multiple annotators may be required to consistently redact information that represents each PHI class. Automated de-identification has the potential to improve annotation quality and reduce annotation time. For instance, using machine-assisted annotation by combining de-identification system outputs used as pre-annotations and an interactive annotation interface to provide annotators with PHI annotations for "curation" rather than manual annotation from "scratch" on raw clinical documents. In order to assess whether machine-assisted annotation improves the reliability and accuracy of the reference standard quality and reduces annotation effort, we conducted an annotation experiment. In this annotation study, we assessed the generalizability of the VA Consortium for Healthcare Informatics Research (CHIR) annotation schema and guidelines applied to a corpus of publicly available clinical documents called MTSamples. Specifically, our goals were to (1) characterize a heterogeneous corpus of clinical documents manually annotated for risk-ranked PHI and other annotation types (clinical eponyms and person relations), (2) evaluate how well annotators apply the CHIR schema to the heterogeneous corpus, (3) compare whether machine-assisted annotation (experiment) improves annotation quality and reduces annotation time compared to manual annotation (control), and (4) assess the change in quality of reference standard coverage with each added annotator's annotations.
authors.affiliation: Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA; VA Salt Lake City Health Care System, Salt Lake City, UT, USA; Department of Internal Medicine, University of Utah, Salt Lake City, UT, USA. Electronic address: brett.south@hsc.utah.edu.
authors.forename: Brett R
authors.initials: BR
authors.lastname: South
authors.affiliation: Department of Biomedical Informatics, University of Pittsburgh, PA, USA; VA Health Care System, San Diego, CA, USA; Division of Biomedical Informatics, University of California San Diego, La Jolla, CA, USA.
authors.forename: Danielle
authors.initials: D
authors.lastname: Mowery
authors.affiliation: VA Salt Lake City Health Care System, Salt Lake City, UT, USA; Department of Internal Medicine, University of Utah, Salt Lake City, UT, USA.
authors.forename: Ying
authors.initials: Y
authors.lastname: Suo
authors.affiliation: VA Salt Lake City Health Care System, Salt Lake City, UT, USA; Department of Internal Medicine, University of Utah, Salt Lake City, UT, USA.
authors.forename: Jianwei
authors.initials: J
authors.lastname: Leng
authors.affiliation: Nuance Communications Inc., Burlington, MA, USA.
authors.forename: Óscar
authors.initials: Ó
authors.lastname: Ferrández
authors.affiliation: Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA; VA Salt Lake City Health Care System, Salt Lake City, UT, USA.
authors.forename: Stephane M
authors.initials: SM
authors.lastname: Meystre
authors.affiliation: Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA; VA Salt Lake City Health Care System, Salt Lake City, UT, USA; Department of Biomedical Informatics, University of Pittsburgh, PA, USA; VA Health Care System, San Diego, CA, USA; Division of Biomedical Informatics, University of California San Diego, La Jolla, CA, USA.
authors.forename: Wendy W
authors.initials: WW
authors.lastname: Chapman
mesh_terms: D057286:Electronic Health Records; D020408:Health Insurance Portability and Accountability Act; D014481:United States; D014584:User-Computer Interface
keywords: Anonymization; Clinical corpora; Confidentiality; De-identification; Electronic health records; Medical informatics; Natural language processing; Patient data privacy