Combining ratings from multiple physician reviewers helped to overcome the uncertainty associated with adverse event classification.
OBJECTIVE
Adverse events (AEs) are poor patient outcomes, resulting from medical care. We performed this study to quantify the misclassification rate obtained using current AE detection methods and to evaluate the effect of combining physician AE ratings.


METHODS
Three physicians independently rated poor patient outcomes. We used latent class analysis to obtain estimates for AE prevalence and reviewer accuracy. These estimates were used as a base case for four simulations of 10,000 cases rated independently by five reviewers. We assessed the effect of AE prevalence, reviewer accuracy, and the number of agreeing reviewers on the probability that cases were correctly classified as an AE.


RESULTS
Reviewer sensitivity and specificity for AE classification were 0.86 and 0.94, respectively. When prevalence was 3%, the positive predictive value that an AE occurred when a single reviewer classified the case as such was 31%, whereas when 2/3 reviewers did so it was 51%. The positive predictive values of ratings for AE occurrence increased with AE prevalence, reviewer accuracy, and the number of reviewers.


CONCLUSIONS
Current methods of AE detection overestimate the risk of AE. Uncertainty regarding the presence of an AE can be overcome by increasing the number of reviews.
Forster
Alan J
AJ
Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada. aforster@ohri.ca
O'Rourke
Keith
K
Shojania
Kaveh G
KG
van Walraven
Carl
C
D003627:Data Interpretation, Statistical; D003955:Diagnostic Tests, Routine / Q000009:adverse effects*; D015987:Effect Modifier, Epidemiologic; D006801:Humans; D018024:Peer Review, Health Care; D010820:Physicians; D012680:Sensitivity and Specificity; D035501:Uncertainty