title: Combining ratings from multiple physician reviewers helped to overcome the uncertainty associated with adverse event classification.
abstract: OBJECTIVE
  Adverse events (AEs) are poor patient outcomes, resulting from medical care. We performed this study to quantify the misclassification rate obtained using current AE detection methods and to evaluate the effect of combining physician AE ratings.
  
  
  METHODS
  Three physicians independently rated poor patient outcomes. We used latent class analysis to obtain estimates for AE prevalence and reviewer accuracy. These estimates were used as a base case for four simulations of 10,000 cases rated independently by five reviewers. We assessed the effect of AE prevalence, reviewer accuracy, and the number of agreeing reviewers on the probability that cases were correctly classified as an AE.
  
  
  RESULTS
  Reviewer sensitivity and specificity for AE classification were 0.86 and 0.94, respectively. When prevalence was 3%, the positive predictive value that an AE occurred when a single reviewer classified the case as such was 31%, whereas when 2/3 reviewers did so it was 51%. The positive predictive values of ratings for AE occurrence increased with AE prevalence, reviewer accuracy, and the number of reviewers.
  
  
  CONCLUSIONS
  Current methods of AE detection overestimate the risk of AE. Uncertainty regarding the presence of an AE can be overcome by increasing the number of reviews.
authors: Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada. aforster@ohri.ca
authors: Alan J
authors: AJ
authors: Forster
authors: Keith
authors: K
authors: O'Rourke
authors: Kaveh G
authors: KG
authors: Shojania
authors: Carl
authors: C
authors: van Walraven
mesh_terms: D003627:Data Interpretation, Statistical; D003955:Diagnostic Tests, Routine / Q000009:adverse effects*; D015987:Effect Modifier, Epidemiologic; D006801:Humans; D018024:Peer Review, Health Care; D010820:Physicians; D012680:Sensitivity and Specificity; D035501:Uncertainty