title: Design and analysis choices for safety surveillance evaluations need to be tuned to the specifics of the hypothesized drug-outcome association.
abstract: We reviewed the results of the Observational Medical Outcomes Research Partnership (OMOP) 2010 Experiment in hopes of finding examples where apparently well-designed drug studies repeatedly produce anomalous findings. OMOP had applied thousands of designs and design parameters to 53 drug-outcome pairs across 10 electronic data resources. Our intent was to use this repository to elucidate some sources of error in observational studies.
  
  
  
  From the 2010 OMOP Experiment, we sought drug-outcome-method combinations (DOMCs) that met consensus design criteria, yet repeatedly produced results contrary to expectation. We set aside DOMCs for which we could not agree on the suitability of the designs, then selected for an in-depth scrutiny one drug-outcome pair analyzed by a seemingly plausible methodological approach, whose results consistently disagreed with the a priori expectation.
  
  
  
  The OMOP "all-by-all" assessment of possible DOMCs yielded many combinations that would not be chosen by researchers as actual study options. Among those that passed a first level of scrutiny, two of seven drug-outcome pairs for which there were plausible research designs had anomalous results. The use of benzodiazepines was unexpectedly associated with acute renal failure and upper gastrointestinal bleeding. We chose the latter as an example for in-depth study. The factitious appearance of a bleeding risk may have been partly driven by an excess of procedures on the first day of treatment. A risk window definition that excluded the first day largely removed the spurious association.
  
  
  
  One cause of reproducible "error" may be repeated failure to tie design choices closely enough to the research question at hand. Copyright © 2016 John Wiley & Sons, Ltd.
authors.affiliation: Reagan-Udall Foundation for the FDA, Innovation in Medical Evidence Development and Surveillance, Washington, DC, USA.|Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MA, USA.
authors.forename: Susan
authors.initials: S
authors.lastname: Gruber
authors.affiliation: Office of Biostatistics, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA.
authors.forename: Aloka
authors.initials: A
authors.lastname: Chakravarty
authors.affiliation: Cardiovascular Health Research Unit, Department of Epidemiology, University of Washington, Seattle, WA, USA.|Group Health Research Institute, Group Health Cooperative, Seattle, WA, USA.
authors.forename: Susan R
authors.initials: SR
authors.lastname: Heckbert
authors.affiliation: Office of Biostatistics, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA.
authors.forename: Mark
authors.initials: M
authors.lastname: Levenson
authors.affiliation: Office of the Center Director, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA.
authors.forename: David
authors.initials: D
authors.lastname: Martin
authors.affiliation: Biostatistics Unit, Group Health Research Institute, Department of Biostatistics, University of Washington, Seattle, WA, USA.
authors.forename: Jennifer C
authors.initials: JC
authors.lastname: Nelson
authors.affiliation: Group Health Research Institute, Group Health Cooperative, Seattle, WA, USA.|Cardiovascular Health Research Unit, Departments of Medicine, Epidemiology, and Health Services, University of Washington, Seattle, WA, USA.
authors.forename: Bruce M
authors.initials: BM
authors.lastname: Psaty
authors.affiliation: Office of Surveillance and Epidemiology, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA.
authors.forename: Simone
authors.initials: S
authors.lastname: Pinheiro
authors.affiliation: RWE Systems, IMS Health, Burlington, MA, USA.
authors.forename: Christian G
authors.initials: CG
authors.lastname: Reich
authors.affiliation: Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MA, USA.
authors.forename: Sengwee
authors.initials: S
authors.lastname: Toh
authors.affiliation: WHISCON, Newton, MA, USA.
authors.forename: Alexander M
authors.initials: AM
authors.lastname: Walker
mesh_terms: D064420:Drug-Related Side Effects and Adverse Reactions / Q000175:diagnosis; D006801:Humans; D064887:Observational Studies as Topic / Q000379:methods* / Q000592:standards; D017063:Outcome Assessment, Health Care / Q000379:methods*; D011358:Product Surveillance, Postmarketing / Q000379:methods*; D015203:Reproducibility of Results; D012107:Research Design
keywords: electronic health records; insurance claim data; medical product safety; monitoring; pharmacoepidemiology