title: The validity and reliability of a signal impact assessment tool.
abstract: The Netherlands Pharmacovigilance Centre Lareb uses an operating Signal Impact Assessment Tool (SIAT) as aid in signal selection. SIAT prioritized signals into one of four categories: strong/moderate signal strength and similarly health impact. Although the SIAT has been used for many years, validity and reliability was never explored.
  
  
  
  The aim of this study is to test the validity and reliability of the operating and weight-adjusted SIAT.
  
  
  
  For validity testing, judgments of a Delphi panel of three pharmacovigilance experts were used as a 'gold standard'. First, the panel judged the weighting of the items included in the SIAT. Then, during two phases, the panel rated 40 signals in one of the four categories. Two researchers scored the signals using the SIAT. Panel judgments were compared with scores for the operating and weight-adjusted SIAT. Inter- and intra-observer variability was also tested. The Cohen's Kappa coefficient (k) was calculated. At least substantial agreement (k > 0.6) was considered to be necessary for an acceptable reliability.
  
  
  
  Validity did not meet predefined criteria: operating and weight-adjusted tool, respectively, k-phase1 = 0.83 and 0.83, k-phase2 = 0.18 and 0.36. Differences were found for signal strength and health impact. Inter- and intra-observer variabilities were good, k of 0.78 and 0.72, respectively.
  
  
  
  The SIAT was found to have an insufficient validity and proper reliability. Although SIAT scores should not be decisive in the decision making process, items included can be used as an aid to decide which signals deserve further investigation. Copyright © 2016 John Wiley & Sons, Ltd.
authors.affiliation: Netherlands Pharmacovigilance Centre Lareb, Hertogenbosch, The Netherlands.|Department of Pharmacy: Pharmacotherapy and Pharmaceutical Care, University of Groningen, Groningen, The Netherlands.
authors.forename: Leàn
authors.initials: L
authors.lastname: Rolfes
authors.affiliation: Department of Pharmacy: Pharmacoepidemiology and Clinical Pharmacology, Utrecht University, Utrecht, The Netherlands.
authors.forename: Judith
authors.initials: J
authors.lastname: Kolfschoten
authors.affiliation: Netherlands Pharmacovigilance Centre Lareb, Hertogenbosch, The Netherlands.|Department of Pharmacy: Pharmacotherapy and Pharmaceutical Care, University of Groningen, Groningen, The Netherlands.
authors.forename: Florence
authors.initials: F
authors.lastname: van Hunsel
authors.affiliation: Netherlands Pharmacovigilance Centre Lareb, Hertogenbosch, The Netherlands.
authors.forename: Eugène
authors.initials: E
authors.lastname: van Puijenbroek
mesh_terms: D016907:Adverse Drug Reaction Reporting Systems / Q000706:statistics & numerical data*; D003657:Decision Making; D003697:Delphi Technique; D064420:Drug-Related Side Effects and Adverse Reactions / Q000453:epidemiology*; D006801:Humans; D009426:Netherlands; D015588:Observer Variation; D060735:Pharmacovigilance; D015203:Reproducibility of Results
keywords: adverse drug reactions (ADRs); pharmacoepidemiology; pharmacovigilance; priority; signal prioritization