Illusions of objectivity and a recommendation for reporting data mining results.
OBJECTIVE
Data mining algorithms (DMAs) are being applied to spontaneous reporting system (SRS) databases in the hope of obtaining timely insights into post-licensure safety data. Some DMAs have been characterized as "objective" screening tools. However, there are numerous available modifiable configuration parameters to choose from, including choice of vendor, that may affect results. Our objective is to compare the data mining results on pre-selected drug-event combinations (DECs) between two commonly used software programs using similar protocols.


METHODS
Two DMAs, using three thresholds, were retrospectively applied to the USFDA safety database through Q2 2005 to a set of eight pre-selected DECs.


RESULTS
Differences between the two vendors were found for the number of cases associated with a signal of disproportionate reporting (SDR), first year of SDRs, and the magnitude of the SDR scores for the selected DECs. These were deemed to be potentially significant for 45.8% (11/24) of the data points.


CONCLUSIONS
The observed differences between vendors could partially be explained by their differing methods of data cleaning and transformation as well as by the specific features of individual algorithms. The choices of vendors and available data mining configurations maximize the exploratory capacity of data mining, but they also raise questions about the claimed objectivity of data mining results and can make data mining exercises susceptible to confirmation bias given the exploratory nature of data mining in pharmacovigilance. When reporting results, the vendor and all data mining configuration details should be specified.
Hauben
Manfred
M
Department of Medicine, New York University School of Medicine, Valhalla, NY, USA.
Reich
Lester
L
Gerrits
Charles M
CM
Younus
Muhammad
M
D016907:Adverse Drug Reaction Reporting Systems / Q000706:statistics & numerical data*; D000465:Algorithms; D003132:Commerce; D003627:Data Interpretation, Statistical; D016208:Databases, Factual; D064420:Drug-Related Side Effects and Adverse Reactions; D006801:Humans; D011358:Product Surveillance, Postmarketing / Q000379:methods* / Q000706:statistics & numerical data; D015203:Reproducibility of Results; D012189:Retrospective Studies; D012984:Software; D014481:United States; D014486:United States Food and Drug Administration