title: A comparison of the empirical performance of methods for a risk identification system.
abstract: BACKGROUND
  Observational healthcare data offer the potential to enable identification of risks of medical products, and the medical literature is replete with analyses that aim to accomplish this objective. A number of established analytic methods dominate the literature but their operating characteristics in real-world settings remain unknown.
  
  
  OBJECTIVE
  To compare the performance of seven methods (new user cohort, case control, self-controlled case series, self-controlled cohort, disproportionality analysis, temporal pattern discovery, and longitudinal gamma poisson shrinker) as tools for risk identification in observational healthcare data.
  
  
  METHODS
  The experiment applied each method to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record).
  
  
  METHODS
  Method performance was evaluated through Area Under the receiver operator characteristics Curve (AUC), bias, mean square error, and confidence interval coverage probability.
  
  
  RESULTS
  Multiple methods offer strong predictive accuracy, with AUC > 0.70 achievable for all outcomes and databases with more than one analytical approach. Self-controlled methods (self-controlled case series, temporal pattern discovery, self-controlled cohort) had higher predictive accuracy than cohort and case-control methods across all databases and outcomes. Methods differed in the expected value and variance of the error distribution. All methods had lower coverage probability than the expected nominal properties.
  
  
  CONCLUSIONS
  Observational healthcare data can inform risk identification of medical product effects on acute liver injury, acute myocardial infarction, acute renal failure and gastrointestinal bleeding. However, effect estimates from all methods require calibration to address inconsistency in method operating characteristics. Further empirical evaluation is required to gauge the generalizability of these findings to other databases and outcomes.
authors.affiliation: Janssen Research and Development LLC, 1125 Trenton-Harbourton Road, Room K30205, PO Box 200, Titusville, NJ, 08560, USA, ryan@omop.org.
authors.forename: Patrick B
authors.initials: PB
authors.lastname: Ryan
authors.forename: Paul E
authors.initials: PE
authors.lastname: Stang
authors.forename: J Marc
authors.initials: JM
authors.lastname: Overhage
authors.forename: Marc A
authors.initials: MA
authors.lastname: Suchard
authors.forename: Abraham G
authors.initials: AG
authors.lastname: Hartzema
authors.forename: William
authors.initials: W
authors.lastname: DuMouchel
authors.forename: Christian G
authors.initials: CG
authors.lastname: Reich
authors.forename: Martijn J
authors.initials: MJ
authors.lastname: Schuemie
authors.forename: David
authors.initials: D
authors.lastname: Madigan
mesh_terms: D019540:Area Under Curve; D016208:Databases, Factual; D064420:Drug-Related Side Effects and Adverse Reactions / Q000175:diagnosis*; D006801:Humans; D012107:Research Design; D018570:Risk Assessment / Q000379:methods*