Do false positive alerts in naïve clinical decision support system lead to false adoption by physicians? A randomized controlled trial.
OBJECTIVE
False positive alerts in patient-safety-related clinical decision support systems (CDSS) are defined as alerts which incorrectly prompt when no-risk patients are encountered. It is an unfavorable condition which may potentially mislead physicians. The aim is to investigate physician responses toward false positive (FP) and true positive (TP) alerts in CDSS for the prevention of contrast-induced nephropathy (CIN).


METHODS
A two-arm cluster randomized controlled trial was conducted in university hospitals. Eligible physicians were randomized to receive alert intervention or no intervention (groups 1 and 2, respectively). The alert system was embedded with a deliberately non-specific risk detection tool in order to generate TP and FP alerts. The naïve alert system would alert the physician to cancel the order regardless of the patient being at-risk or not at-risk. CIN risk was stratified as at-risk and no-risk according to a patient's pre-existing renal function. Contrast imaging order-cancellation rate was measured as primary outcome.


RESULTS
3802 contrast-enhanced examination orders from 66 physicians were analyzed. Demographic data and risk distributions of patients were similar and well-balanced between two groups. In the intervention group, a total of 1892 alerts were generated (332 TP alerts and 1560 FP alerts). Order-cancellation rates were 5.1% versus 1.4% in groups 1 and 2 for at-risk patients (relative risk [RR] = 3.69) from TP alerts, and 1.0% versus 1.4% for no-risk patients (RR = 0.71) from FP alerts. Using generalized linear model with generalized estimating equation, the FP alerts had no order-cancellation effect when compared to the control arm (adjusted RR = 0.69; 95%CI, 0.36-1.32). The TP alerts had a larger order-cancellation effect than that of the control arm (adjusted RR = 2.95; 95%CI, 0.94-9.27), which revealed a marginal trend toward significance. However, the effect was not statistically significant (adjusted RR = 1.24; 95%CI, 0.71-2.18) if TP and FP alerts were mixed.


CONCLUSIONS
Physicians are not likely to adopt recommendations provided by false positive alerts in patient-safety-related CDSS. If reporting only the adoption rate of CDSS as a whole without differentiating between TP and FP alerts, the effects of TP and FP alerts will be mixed, and thus, will lead to an underestimation of system effectiveness.
Institute of Biomedical Informatics, National Yang-Ming University, Taipei, Taiwan; Division of Urology, Department of Surgery, Far Eastern Memorial Hospital, New Taipei, Taiwan.
Chung-You
CY
Tsai
Institute of Epidemiology and Preventive Medicine, College of Public Health, National Taiwan University, Taipei, Taiwan; Graduate Institute of Biostatistics, China Medical University, Taichung, Taiwan.
Shi-Heng
SH
Wang
Office of International Cooperation, Ministry of Health and Welfare, Taiwan; College of Medicine Science and Technology (CoMST), Taipei Medical University, Taiwan; International Center for Health Information Technology (ICHIT), Taipei Medical University, Taiwan.
Min-Huei
MH
Hsu
College of Medicine Science and Technology (CoMST), Taipei Medical University, Taiwan; International Center for Health Information Technology (ICHIT), Taipei Medical University, Taiwan; Department of Dermatology, Wan Fang Hospital, Taipei, Taiwan. Electronic address: jack@tmu.edu.tw.
Yu-Chuan Jack
YC
Li
D000367:Age Factors; D000368:Aged; D000369:Aged, 80 and over; D020000:Decision Support Systems, Clinical; D005260:Female; D005919:Glomerular Filtration Rate; D006801:Humans; D008297:Male; D008875:Middle Aged; D013624:Taiwan
Alert system; Clinical decision support system; Contrast-induced nephropathy; False positive alert; Patient safety; Randomized controlled trial