title: Development of a Standardized Rating Tool for Drug Alerts to Reduce Information Overload.
abstract: BACKGROUND
  A well-known problem in current clinical decision support systems (CDSS) is the high number of alerts, which are often medically incorrect or irrelevant. This may lead to the so-called alert fatigue, an overriding of alerts, including those that are clinically relevant, and underuse of CDSS in general.
  
  
  OBJECTIVE
  The aim of our study was to develop and to apply a standardized tool that allows its users to evaluate the quality of system-generated drug alerts. The users' ratings can subsequently be used to derive recommendations for developing a filter function to reduce irrelevant alerts.
  
  
  METHODS
  We developed a rating tool for drug alerts and performed a web-based evaluation study that also included a user review of alerts. In this study the following categories were evaluated: "data linked correctly", "medically correct", "action required", "medication change", "critical alert", "information gained" and "show again". For this purpose, 20 anonymized clinical cases were randomly selected and displayed in our customized CDSS research prototype, which used the summary of product characteristics (SPC) for alert generation. All the alerts that were provided were evaluated by 13 physicians. The users' ratings were used to derive a filtering algorithm to reduce overalerting.
  
  
  RESULTS
  In total, our CDSS research prototype generated 399 alerts. In 98 % of all alerts, medication data were rated as linked correctly to drug information; in 93 %, the alerts were assessed as "medically correct"; 19.5 % of all alerts were rated as "show again". The interrater-agreement was, on average, 68.4 %. After the application of our filtering algorithm, the rate of alerts that should be shown again decreased to 14.8 %.
  
  
  CONCLUSIONS
  The new standardized rating tool supports a standardized feedback of user-perceived clinical relevance of CDSS alerts. Overall, the results indicated that physicians may consider the majority of alerts formally correct but clinically irrelevant and override them. Filtering may help to reduce overalerting and increase the specificity of a CDSS.
authors: Barbara
authors: B
authors: Pfistermeister
authors: Brita
authors: B
authors: Sedlmayr
authors: Andrius
authors: A
authors: Patapovas
authors: Gerald
authors: G
authors: Suttner
authors: Ozan
authors: O
authors: Tektas
authors: Aleksey
authors: A
authors: Tarkhov
authors: Johannes
authors: J
authors: Kornhuber
authors: Martin F
authors: MF
authors: Fromm
authors: Thomas
authors: T
authors: Bürkle
authors: Hans-Ulrich
authors: HU
authors: Prokosch
authors: Prof. Dr. med. Renke Maas, Institute of Experimental and Clinical Pharmacology and Toxicology, Friedrich-Alexander-Universität Erlangen-Nürnberg, Fahrstr. 17, 91054 Erlangen, Germany, E-mail: renke.maas@fau.de.
authors: Renke
authors: R
authors: Maas
mesh_terms: D000328:Adult; D000465:Algorithms; D020000:Decision Support Systems, Clinical; D004347:Drug Interactions; D006801:Humans; D012015:Reference Standards; D014584:User-Computer Interface
keywords: Clinical decision support systems; assessment tool; evaluation study; medication alert system