title: Accuracy and generalizability of using automated methods for identifying adverse events from electronic health record data: a validation study protocol.
abstract: Adverse events (AEs) in acute care hospitals are frequent and associated with significant morbidity, mortality, and costs. Measuring AEs is necessary for quality improvement and benchmarking purposes, but current detection methods lack in accuracy, efficiency, and generalizability. The growing availability of electronic health records (EHR) and the development of natural language processing techniques for encoding narrative data offer an opportunity to develop potentially better methods. The purpose of this study is to determine the accuracy and generalizability of using automated methods for detecting three high-incidence and high-impact AEs from EHR data: a) hospital-acquired pneumonia, b) ventilator-associated event and, c) central line-associated bloodstream infection.
  
  
  
  This validation study will be conducted among medical, surgical and ICU patients admitted between 2013 and 2016 to the Centre hospitalier universitaire de Sherbrooke (CHUS) and the McGill University Health Centre (MUHC), which has both French and English sites. A random 60% sample of CHUS patients will be used for model development purposes (cohort 1, development set). Using a random sample of these patients, a reference standard assessment of their medical chart will be performed. Multivariate logistic regression and the area under the curve (AUC) will be employed to iteratively develop and optimize three automated AE detection models (i.e., one per AE of interest) using EHR data from the CHUS. These models will then be validated on a random sample of the remaining 40% of CHUS patients (cohort 1, internal validation set) using chart review to assess accuracy. The most accurate models developed and validated at the CHUS will then be applied to EHR data from a random sample of patients admitted to the MUHC French site (cohort 2) and English site (cohort 3)-a critical requirement given the use of narrative data -, and accuracy will be assessed using chart review. Generalizability will be determined by comparing AUCs from cohorts 2 and 3 to those from cohort 1.
  
  
  
  This study will likely produce more accurate and efficient measures of AEs. These measures could be used to assess the incidence rates of AEs, evaluate the success of preventive interventions, or benchmark performance across hospitals.
authors.affiliation: School of Nursing, Faculty of Medicine and Health Sciences, University of Sherbrooke, 3001, 12e Avenue Nord, Sherbrooke, QC, J1H 5N4, Canada. Christian.Rochefort@usherbrooke.ca.|Centre de recherche de l'Hôpital Charles-LeMoyne, University of Sherbrooke-Campus Longueuil, 150 Place Charles-LeMoyne, Longueuil, QC, J4K 0A8, Canada. Christian.Rochefort@usherbrooke.ca.|Department of Epidemiology, Biostatics and Occupational Health, Faculty of Medicine, McGill University, Purvis Hall, 1020 Pine Avenue West, Montreal, QC, H3A 1A2, Canada. Christian.Rochefort@usherbrooke.ca.
authors.forename: Christian M
authors.initials: CM
authors.lastname: Rochefort
authors.affiliation: Department of Epidemiology, Biostatics and Occupational Health, Faculty of Medicine, McGill University, Purvis Hall, 1020 Pine Avenue West, Montreal, QC, H3A 1A2, Canada.
authors.forename: David L
authors.initials: DL
authors.lastname: Buckeridge
authors.affiliation: School of Nursing, Faculty of Medicine and Health Sciences, University of Sherbrooke, 3001, 12e Avenue Nord, Sherbrooke, QC, J1H 5N4, Canada.
authors.forename: Andréanne
authors.initials: A
authors.lastname: Tanguay
authors.affiliation: Department of Quality, Patient Safety and Performance, McGill University Health Centre, 2155 Guy Street, Montreal, QC, H3H 2R9, Canada.|Ingram School of Nursing, McGill University, Wilson Hall, 3506 University Street, Montreal, QC, H3A 2A7, Canada.
authors.forename: Alain
authors.initials: A
authors.lastname: Biron
authors.affiliation: Department of Anesthesiology, Faculty of Medicine and Health Sciences, University of Sherbrooke and Centre hospitalier universitaire de Sherbrooke, 3001, 12e Avenue Nord, Sherbrooke, QC, J1H 5N4, Canada.
authors.forename: Frédérick
authors.initials: F
authors.lastname: D'Aragon
authors.affiliation: Faculty of Sciences, Department of Informatics, University of Sherbrooke, 2500 Boulevard de l'Université, Sherbrooke, QC, J1K 2R1, Canada.
authors.forename: Shengrui
authors.initials: S
authors.lastname: Wang
authors.affiliation: Department of Diagnostic Radiology, McGill University and McGill University Health Centre, 1650 Cedar Avenue, Montreal, QC, H3G 1A4, Canada.
authors.forename: Benoit
authors.initials: B
authors.lastname: Gallix
authors.affiliation: Department of Microbiology and Infectious Diseases, University of Sherbrooke and Centre hospitalier universitaire de Sherbrooke, 3001, 12e Avenue Nord, Sherbrooke, QC, J1H 5N4, Canada.
authors.forename: Louis
authors.initials: L
authors.lastname: Valiquette
authors.affiliation: School of Nursing, Faculty of Medicine and Health Sciences, University of Sherbrooke, 3001, 12e Avenue Nord, Sherbrooke, QC, J1H 5N4, Canada.
authors.forename: Li-Anne
authors.initials: LA
authors.lastname: Audet
authors.affiliation: Department of Internal Medicine, McGill University and McGill University Health Centre, 1650 Cedar Avenue, Montreal, QC, H3G 1A4, Canada.
authors.forename: Todd C
authors.initials: TC
authors.lastname: Lee
authors.affiliation: Department of Internal Medicine, McGill University and McGill University Health Centre, 1650 Cedar Avenue, Montreal, QC, H3G 1A4, Canada.
authors.forename: Dev
authors.initials: D
authors.lastname: Jayaraman
authors.affiliation: Department of Quality, Evaluation, Performance and Ethics, Centre hospitalier universitaire de Sherbrooke, 3001, 12e Avenue Nord, Sherbrooke, QC, J1H 5N4, Canada.
authors.forename: Bruno
authors.initials: B
authors.lastname: Petrucci
authors.affiliation: Department of Quality, Patient Safety and Performance, McGill University Health Centre, 2155 Guy Street, Montreal, QC, H3H 2R9, Canada.
authors.forename: Patricia
authors.initials: P
authors.lastname: Lefebvre
mesh_terms: D002405:Catheterization, Central Venous / Q000009:adverse effects*; D003428:Cross Infection / Q000453:epidemiology*; D057286:Electronic Health Records / Q000706:statistics & numerical data; D005260:Female; D006760:Hospitalization / Q000706:statistics & numerical data; D006761:Hospitals; D006801:Humans; D015994:Incidence; D008297:Male; D009323:Natural Language Processing; D011014:Pneumonia / Q000453:epidemiology; D058996:Quality Improvement; D012121:Respiration, Artificial / Q000009:adverse effects*
keywords: Acute care hospital; Adverse events; Automated detection; Data warehouse; Electronic health record; Natural language processing; Patient safety