title: Assessing severity, immediacy, and ideal setting in ED patients: a pilot study on interrater reliability.
abstract: OBJECTIVE
  We conducted a pilot study to test the interrater reliability of emergency department (ED) physician assessments of 3 ED visit attributes-severity, immediacy, and ideal setting, with the long-term goal of developing a novel ED categorization system.
  
  
  METHODS
  Using 2010 National Hospital Ambulatory Care Survey data, we randomly selected 300 ED patient records for review by 6 emergency medicine physicians. Each record was assessed by 2 physicians for severity and immediacy using a 7-point scale; "ideal" setting was chosen among 6 possible settings. κ-Weighted and unweighted-and interclass correlation coefficients were used to test interrater agreement.
  
  
  RESULTS
  For severity, immediacy, and ideal setting, there was "fair" agreement in assessments with a weighted κ of 0.33 (95% confidence interval [CI], 0.27-0.40), 0.30 (95% CI, 0.23-0.36), and nonweighted κ of 0.28 (95% CI, 0.21-0.34), respectively. When both raters were "very certain" about their assessments, weighted κ increased to 0.42 (95% CI, 0.34-0.51) for severity and 0.35 (95% CI, 0.27-0.44) for immediacy. Interclass correlation coefficients showed similar results. There was agreement on ideal setting for 162 (54%) of 300 scenarios. Scenarios with poor agreement on ideal setting in general involved care for nonspecific symptoms rather than specific diagnoses.
  
  
  CONCLUSIONS
  Rater agreement among ED physicians when assessing clinical data on specific ED visits was fair for severity and immediacy ratings. Raters agreed on ideal treatment settings half the time. In general, there was greater agreement when a specific diagnosis was found rather than negative workups for symptoms. This demonstrates a validity issue when it comes to developing and using categorization systems for ED visits and assessing setting appropriateness.
authors.affiliation: George Washington School of Medicine and Health Sciences, Washington, DC. Electronic address: MichaelMcMullen@gwu.edu.
authors.forename: Michael E
authors.initials: ME
authors.lastname: McMullen
authors.affiliation: George Washington School of Medicine and Health Sciences, Washington, DC; Office of Clinical Practice Innovation, 2100 Pennsylvania, Washington, DC 20037. Electronic address: Pinesj@gwu.edu.
authors.forename: Jesse M
authors.initials: JM
authors.lastname: Pines
mesh_terms: D000328:Adult; D001291:Attitude of Health Personnel; D003430:Cross-Sectional Studies; D004636:Emergency Service, Hospital; D005260:Female; D006801:Humans; D008297:Male; D008875:Middle Aged; D015588:Observer Variation; D010865:Pilot Projects; D015203:Reproducibility of Results; D012720:Severity of Illness Index; D014218:Triage