title: Empirical assessment of methods for risk identification in healthcare data: results from the experiments of the Observational Medical Outcomes Partnership.
abstract: BACKGROUND
  Expanded availability of observational healthcare data (both administrative claims and electronic health records) has prompted the development of statistical methods for identifying adverse events associated with medical products, but the operating characteristics of these methods when applied to the real-world data are unknown.
  
  
  METHODS
  We studied the performance of eight analytic methods for estimating of the strength of association-relative risk (RR) and associated standard error of 53 drug-adverse event outcome pairs, both positive and negative controls. The methods were applied to a network of ten observational healthcare databases, comprising over 130 million lives. Performance measures included sensitivity, specificity, and positive predictive value of methods at RR thresholds achieving statistical significance of p < 0.05 or p < 0.001 and with absolute threshold RR > 1.5, as well as threshold-free measures such as area under receiver operating characteristic curve (AUC).
  
  
  RESULTS
  Although no specific method demonstrated superior performance, the aggregate results provide a benchmark and baseline expectation for risk identification method performance. At traditional levels of statistical significance (RR > 1, p < 0.05), all methods have a false positive rate >18%, with positive predictive value <38%. The best predictive model, high-dimensional propensity score, achieved an AUC  =  0.77. At 50% sensitivity, false positive rate ranged from 16% to 30%. At 10% false positive rate, sensitivity of the methods ranged from 9% to 33%.
  
  
  CONCLUSIONS
  Systematic processes for risk identification can provide useful information to supplement an overall safety assessment, but assessment of methods performance suggests a substantial chance of identifying false positive associations.
authors.affiliation: Johnson & Johnson Pharmaceutical Research and Development LLC, Titusville, NJ 08560, USA. ryan@omop.org
authors.forename: Patrick B
authors.initials: PB
authors.lastname: Ryan
authors.forename: David
authors.initials: D
authors.lastname: Madigan
authors.forename: Paul E
authors.initials: PE
authors.lastname: Stang
authors.forename: J Marc
authors.initials: JM
authors.lastname: Overhage
authors.forename: Judith A
authors.initials: JA
authors.lastname: Racoosin
authors.forename: Abraham G
authors.initials: AG
authors.lastname: Hartzema
mesh_terms: D015984:Causality; D064420:Drug-Related Side Effects and Adverse Reactions; D057286:Electronic Health Records / Q000706:statistics & numerical data*; D006801:Humans; D007345:Insurance Claim Review / Q000706:statistics & numerical data; D017891:Pharmacoepidemiology / Q000379:methods* / Q000706:statistics & numerical data; D011358:Product Surveillance, Postmarketing / Q000379:methods* / Q000706:statistics & numerical data; D018570:Risk Assessment / Q000379:methods