title: A reference standard for evaluation of methods for drug safety signal detection using electronic healthcare record databases.
abstract: BACKGROUND
  The growing interest in using electronic healthcare record (EHR) databases for drug safety surveillance has spurred development of new methodologies for signal detection. Although several drugs have been withdrawn postmarketing by regulatory authorities after scientific evaluation of harms and benefits, there is no definitive list of confirmed signals (i.e. list of all known adverse reactions and which drugs can cause them). As there is no true gold standard, prospective evaluation of signal detection methods remains a challenge.
  
  
  OBJECTIVE
  Within the context of methods development and evaluation in the EU-ADR Project (Exploring and Understanding Adverse Drug Reactions by integrative mining of clinical records and biomedical knowledge), we propose a surrogate reference standard of drug-adverse event associations based on existing scientific literature and expert opinion.
  
  
  METHODS
  The reference standard was constructed for ten top-ranked events judged as important in pharmacovigilance. A stepwise approach was employed to identify which, among a list of drug-event associations, are well recognized (known positive associations) or highly unlikely ('negative controls') based on MEDLINE-indexed publications, drug product labels, spontaneous reports made to the WHO's pharmacovigilance database, and expert opinion. Only drugs with adequate exposure in the EU-ADR database network (comprising ≈60 million person-years of healthcare data) to allow detection of an association were considered. Manual verification of positive associations and negative controls was independently performed by two experts proficient in clinical medicine, pharmacoepidemiology and pharmacovigilance. A third expert adjudicated equivocal cases and arbitrated any disagreement between evaluators.
  
  
  RESULTS
  Overall, 94 drug-event associations comprised the reference standard, which included 44 positive associations and 50 negative controls for the ten events of interest: bullous eruptions; acute renal failure; anaphylactic shock; acute myocardial infarction; rhabdomyolysis; aplastic anaemia/pancytopenia; neutropenia/agranulocytosis; cardiac valve fibrosis; acute liver injury; and upper gastrointestinal bleeding. For cardiac valve fibrosis, there was no drug with adequate exposure in the database network that satisfied the criteria for a positive association.
  
  
  CONCLUSIONS
  A strategy for the construction of a reference standard to evaluate signal detection methods that use EHR has been proposed. The resulting reference standard is by no means definitive, however, and should be seen as dynamic. As knowledge on drug safety evolves over time and new issues in drug safety arise, this reference standard can be re-evaluated.
authors: Department of Medical Informatics, Erasmus Medical Center, Postbus 2040, 3000, CA, Rotterdam, The Netherlands. p.coloma@erasmusmc.nl
authors: Preciosa M
authors: PM
authors: Coloma
authors: Paul
authors: P
authors: Avillach
authors: Francesco
authors: F
authors: Salvo
authors: Martijn J
authors: MJ
authors: Schuemie
authors: Carmen
authors: C
authors: Ferrajolo
authors: Antoine
authors: A
authors: Pariente
authors: Annie
authors: A
authors: Fourrier-Réglat
authors: Mariam
authors: M
authors: Molokhia
authors: Vaishali
authors: V
authors: Patadia
authors: Johan
authors: J
authors: van der Lei
authors: Miriam
authors: M
authors: Sturkenboom
authors: Gianluca
authors: G
authors: Trifirò
mesh_terms: D016907:Adverse Drug Reaction Reporting Systems / Q000458:organization & administration / Q000706:statistics & numerical data; D016208:Databases, Factual / Q000706:statistics & numerical data; D064420:Drug-Related Side Effects and Adverse Reactions; D057286:Electronic Health Records / Q000706:statistics & numerical data*; D006801:Humans; D060735:Pharmacovigilance; D011358:Product Surveillance, Postmarketing / Q000379:methods*; D012015:Reference Standards