title: Evaluating performance of risk identification methods through a large-scale simulation of observational data.
abstract: BACKGROUND
  There has been only limited evaluation of statistical methods for identifying safety risks of drug exposure in observational healthcare data. Simulations can support empirical evaluation, but have not been shown to adequately model the real-world phenomena that challenge observational analyses.
  
  
  OBJECTIVE
  To design and evaluate a probabilistic framework (OSIM2) for generating simulated observational healthcare data, and to use this data for evaluating the performance of methods in identifying associations between drug exposure and health outcomes of interest.
  
  
  METHODS
  Seven observational designs, including case-control, cohort, self-controlled case series, and self-controlled cohort design were applied to 399 drug-outcome scenarios in 6 simulated datasets with no effect and injected relative risks of 1.25, 1.5, 2, 4, and 10, respectively.
  
  
  METHODS
  Longitudinal data for 10 million simulated patients were generated using a model derived from an administrative claims database, with associated demographics, periods of drug exposure derived from pharmacy dispensings, and medical conditions derived from diagnoses on medical claims.
  
  
  METHODS
  Simulation validation was performed through descriptive comparison with real source data. Method performance was evaluated using Area Under ROC Curve (AUC), bias, and mean squared error.
  
  
  RESULTS
  OSIM2 replicates prevalence and types of confounding observed in real claims data. When simulated data are injected with relative risks (RR) ≥ 2, all designs have good predictive accuracy (AUC > 0.90), but when RR < 2, no methods achieve 100 % predictions. Each method exhibits a different bias profile, which changes with the effect size.
  
  
  CONCLUSIONS
  OSIM2 can support methodological research. Results from simulation suggest method operating characteristics are far from nominal properties.
authors.affiliation: Janssen Research and Development LLC, 1125 Trenton-Harbourton Road, Room K30205, PO Box 200, Titusville, NJ, 08560, USA, ryan@omop.org.
authors.forename: Patrick B
authors.initials: PB
authors.lastname: Ryan
authors.forename: Martijn J
authors.initials: MJ
authors.lastname: Schuemie
mesh_terms: D000328:Adult; D003198:Computer Simulation; D016208:Databases, Factual; D064420:Drug-Related Side Effects and Adverse Reactions / Q000175:diagnosis*; D005260:Female; D006801:Humans; D008297:Male; D064887:Observational Studies as Topic; D012107:Research Design; D018570:Risk Assessment / Q000379:methods*