title: A time-indexed reference standard of adverse drug reactions.
abstract: Undetected adverse drug reactions (ADRs) pose a major burden on the health system. Data mining methodologies designed to identify signals of novel ADRs are of deep importance for drug safety surveillance. The development and evaluation of these methodologies requires proper reference benchmarks. While progress has recently been made in developing such benchmarks, our understanding of the performance characteristics of the data mining methodologies is limited because existing benchmarks do not support prospective performance evaluations. We address this shortcoming by providing a reference standard to support prospective performance evaluations. The reference standard was systematically curated from drug labeling revisions, such as new warnings, which were issued and communicated by the US Food and Drug Administration in 2013. The reference standard includes 62 positive test cases and 75 negative controls, and covers 44 drugs and 38 events. We provide usage guidance and empirical support for the reference standard by applying it to analyze two data sources commonly mined for drug safety surveillance.
authors: Center for Biomedical Informatics Research, Stanford University, Stanford, California 94305, USA.
authors: Rave
authors: R
authors: Harpaz
authors: Center for Biomedical Informatics Research, Stanford University, Stanford, California 94305, USA.
authors: David
authors: D
authors: Odgers
authors: Center for Biomedical Informatics Research, Stanford University, Stanford, California 94305, USA.
authors: Greg
authors: G
authors: Gaskin
authors: Oracle Health Sciences, Bedford, Massachusetts 01730, USA.
authors: William
authors: W
authors: DuMouchel
authors: National Library of Medicine, NIH, Bethesda, Maryland 20894, USA.
authors: Rainer
authors: R
authors: Winnenburg
authors: National Library of Medicine, NIH, Bethesda, Maryland 20894, USA.
authors: Olivier
authors: O
authors: Bodenreider
authors: National Library of Medicine, NIH, Bethesda, Maryland 20894, USA.
authors: Anna
authors: A
authors: Ripple
authors: U.S. FDA, Silver Spring, Maryland 20993, USA.
authors: Ana
authors: A
authors: Szarfman
authors: U.S. FDA, Silver Spring, Maryland 20993, USA.
authors: Alfred
authors: A
authors: Sorbello
authors: Microsoft Research, Redmond, Washington 98052, USA.
authors: Eric
authors: E
authors: Horvitz
authors: Microsoft Research, Redmond, Washington 98052, USA.
authors: Ryen W
authors: RW
authors: White
authors: Center for Biomedical Informatics Research, Stanford University, Stanford, California 94305, USA.
authors: Nigam H
authors: NH
authors: Shah
mesh_terms: D016907:Adverse Drug Reaction Reporting Systems / Q000592:standards*; D057225:Data Mining; D004341:Drug Evaluation / Q000592:standards; D004348:Drug Labeling / Q000592:standards; D064420:Drug-Related Side Effects and Adverse Reactions; D006801:Humans; D016239:MEDLINE; D012015:Reference Standards; D013997:Time Factors; D014481:United States; D014486:United States Food and Drug Administration
references: J Am Med Inform Assoc. 2013 May 1;20(3):446-52
references: 23195749
references: J Am Med Inform Assoc. 2011 Sep-Oct;18(5):668-74
references: 21546507
references: Drug Saf. 2013 Jan;36(1):13-23
references: 23315292
references: Drug Saf. 2013 Oct;36 Suppl 1:S33-47
references: 24166222
references: Drug Saf. 2011 Apr 1;34(4):267-71
references: 21417499
references: Clin Pharmacol Ther. 2013 Jun;93(6):539-46
references: 23571771
references: J Am Med Inform Assoc. 2013 May 1;20(3):404-8
references: 23467469
references: J Am Med Inform Assoc. 2013 May 1;20(3):413-9
references: 23118093
references: Clin Pharmacol Ther. 2014 Aug;96(2):239-46
references: 24713590
references: Clin Pharmacol Ther. 2012 Jun;91(6):1010-21
references: 22549283
references: J Gen Intern Med. 2003 Jan;18(1):57-60
references: 12534765
references: Clin Pharmacol Ther. 2013 Jun;93(6):474-5
references: 23689213
references: Clin Pharmacol Ther. 2009 Mar;85(3):259-68
references: 19177064
references: Drug Saf. 2014 Aug;37(8):557-67
references: 24985530
references: N Engl J Med. 2009 Aug 13;361(7):645-7
references: 19635947
references: Pharmacoepidemiol Drug Saf. 2011 Jan;20(1):1-11
references: 21182150
references: Drug Saf. 2014 Jul;37(7):471-9
references: 24895178
references: Drug Saf. 2014 Sep;37(9):655-9
references: 25005708
references: Clin Pharmacol Ther. 2013 Jun;93(6):547-55
references: 23571773
references: J Am Med Inform Assoc. 2011 Dec;18 Suppl 1:i73-80
references: 21946238
references: Drug Saf. 2002;25(6):381-92
references: 12071774
references: JAMA. 1998 Apr 15;279(15):1200-5
references: 9555760
references: Ann Intern Med. 2010 Nov 2;153(9):600-6
references: 21041580
references: JAMA. 1997 Jan 22-29;277(4):301-6
references: 9002492