title: Comparison of statistical signal detection methods within and across spontaneous reporting databases.
abstract: BACKGROUND
  Most pharmacovigilance departments maintain a system to identify adverse drug reactions (ADRs) through analysis of spontaneous reports. The signal detection algorithms (SDAs) and the nature of the reporting databases vary between operators and it is unclear whether any algorithm can be expected to provide good performance in a wide range of environments.
  
  
  OBJECTIVE
  The objective of this study was to compare the performance of commonly used algorithms across spontaneous reporting databases operated by pharmaceutical companies and national and international pharmacovigilance organisations.
  
  
  METHODS
  220 products were chosen and a reference set of ADRs was compiled. Within four company, one national and two international databases, 15 SDAs based on five disproportionality methods were tested. Signals of disproportionate reporting (SDRs) were calculated at monthly intervals and classified by comparison with the reference set. These results were summarised as sensitivity and precision for each algorithm in each database.
  
  
  RESULTS
  Different algorithms performed differently between databases but no method dominated all others. Performance was strongly dependent on the thresholds used to define a statistical signal. However, the different disproportionality statistics did not influence the achievable performance. The relative performance of two algorithms was similar in different databases. Over the lifetime of a product there is a reduction in precision for any method.
  
  
  CONCLUSIONS
  In designing signal detection systems, careful consideration should be given to the criteria that are used to define an SDR. The choice of disproportionality statistic does not appreciably affect the achievable range of signal detection performance and so this can primarily be based on ease of implementation, interpretation and minimisation of computing resources. The changes in sensitivity and precision obtainable by replacing one algorithm with another are predictable. However, the absolute performance of a method is specific to the database and is best assessed directly on that database. New methods may be required to gain appreciable improvements.
authors.affiliation: European Medicines Agency, 7 Westferry Circus, Canary Wharf, London, E14 4HB, UK.
authors.forename: Gianmario
authors.initials: G
authors.lastname: Candore
authors.forename: Kristina
authors.initials: K
authors.lastname: Juhlin
authors.forename: Katrin
authors.initials: K
authors.lastname: Manlik
authors.forename: Bharat
authors.initials: B
authors.lastname: Thakrar
authors.forename: Naashika
authors.initials: N
authors.lastname: Quarcoo
authors.forename: Suzie
authors.initials: S
authors.lastname: Seabroke
authors.forename: Antoni
authors.initials: A
authors.lastname: Wisniewski
authors.forename: Jim
authors.initials: J
authors.lastname: Slattery
mesh_terms: D016907:Adverse Drug Reaction Reporting Systems / Q000706:statistics & numerical data*; D000465:Algorithms; D016208:Databases, Factual / Q000706:statistics & numerical data; D064420:Drug-Related Side Effects and Adverse Reactions / Q000453:epidemiology; D006801:Humans; D060735:Pharmacovigilance; D012680:Sensitivity and Specificity