title: Challenges in the design and analysis of sequentially monitored postmarket safety surveillance evaluations using electronic observational health care data.
abstract: OBJECTIVE
  Many challenges arise when conducting a sequentially monitored medical product safety surveillance evaluation using observational electronic data captured during routine care. We review existing sequential approaches for potential use in this setting, including a continuous sequential testing method that has been utilized within the Vaccine Safety Datalink (VSD) and group sequential methods, which are used widely in randomized clinical trials.
  
  
  METHODS
  Using both simulated data and preliminary data from an ongoing VSD evaluation, we discuss key sequential design considerations, including sample size and duration of surveillance, shape of the signaling threshold, and frequency of interim testing.
  
  
  CONCLUSIONS
  All designs control the overall Type 1 error rate across all tests performed, but each yields different tradeoffs between the probability and timing of true and false positive signals. Designs tailored to monitor efficacy outcomes in clinical trials have been well studied, but less consideration has been given to optimizing design choices for observational safety settings, where the hypotheses, population, prevalence and severity of the outcomes, implications of signaling, and costs of false positive and negative findings are very different. Analytic challenges include confounding, missing and partially accrued data, high misclassification rates for outcomes presumptively defined using diagnostic codes, and unpredictable changes in dynamically accessed data over time (e.g., differential product uptake). Many of these factors influence the variability of the adverse events under evaluation and, in turn, the probability of committing a Type 1 error. Thus, to ensure proper Type 1 error control, planned sequential thresholds should be adjusted over time to account for these issues.
authors.affiliation: Biostatistics Unit, Group Health Research Institute, Seattle, WA 98101, USA. nelson.jl@ghc.org
authors.forename: Jennifer C
authors.initials: JC
authors.lastname: Nelson
authors.forename: Andrea J
authors.initials: AJ
authors.lastname: Cook
authors.forename: Onchee
authors.initials: O
authors.lastname: Yu
authors.forename: Clara
authors.initials: C
authors.lastname: Dominguez
authors.forename: Shanshan
authors.initials: S
authors.lastname: Zhao
authors.forename: Sharon K
authors.initials: SK
authors.lastname: Greene
authors.forename: Bruce H
authors.initials: BH
authors.lastname: Fireman
authors.forename: Steven J
authors.initials: SJ
authors.lastname: Jacobsen
authors.forename: Eric S
authors.initials: ES
authors.lastname: Weintraub
authors.forename: Lisa A
authors.initials: LA
authors.lastname: Jackson
mesh_terms: D016907:Adverse Drug Reaction Reporting Systems / Q000706:statistics & numerical data; D015986:Confounding Factors, Epidemiologic; D016208:Databases, Factual; D057286:Electronic Health Records / Q000706:statistics & numerical data*; D015340:Epidemiologic Research Design; D006801:Humans; D011358:Product Surveillance, Postmarketing / Q000379:methods*; D016032:Randomized Controlled Trials as Topic / Q000379:methods; D014612:Vaccines / Q000009:adverse effects
chemical_list: D014612:Vaccines