title: Interrater reliability and accuracy of clinicians and trained research assistants performing prospective data collection in emergency department patients with potential acute coronary syndrome.
abstract: OBJECTIVE
  Clinical research requires high-quality data collection. Data collected at the emergency department evaluation is generally considered more precise than data collected through chart abstraction but is cumbersome and time consuming. We test whether trained research assistants without a medical background can obtain clinical research data as accurately as physicians. We hypothesize that they would be at least as accurate because they would not be distracted by clinical requirements.
  
  
  METHODS
  We conducted a prospective comparative study of 33 trained research assistants and 39 physicians (35 residents) to assess interrater reliability with respect to guideline-recommended clinical research data. Immediately after the research assistant and clinician evaluation, the data were compared by a tiebreaker third person who forced the patient to choose one of the 2 answers as the correct one when responses were discordant. Crude percentage agreement and interrater reliability were assessed (kappa statistic).
  
  
  RESULTS
  One hundred forty-three patients were recruited (mean age 50.7 years; 47% female patients). Overall, the median agreement was 81% (interquartile range [IQR] 73% to 92%) and interrater reliability was fair (kappa value 0.36 [IQR 0.26 to 0.52]) but varied across categories of data: cardiac risk factors (median 86% [IQR 81% to 93%]; median 0.69 [IQR 0.62 to 0.83]), other cardiac history (median 93% [IQR 79% to 95%]; median 0.56 [IQR 0.29 to 0.77]), pain location (median 92% [IR 86% to 94%]; median 0.37 [IQR 0.25 to 0.29]), radiation (median 86% [IQR 85% to 87%]; median 0.37 [IQR 0.26 to 0.42]), quality (median 85% [IQR 75% to 94%]; median 0.29 [IQR 0.23 to 0.40]), and associated symptoms (median 74% [IQR 65% to 78%]; median 0.28 [IQR 0.20 to 0.40]). When discordant information was obtained, the research assistant was more often correct (median 64% [IQR 53% to 72%]).
  
  
  CONCLUSIONS
  The relatively fair interrater reliability observed in our study is consistent with previous studies evaluating interrater reliability for cardiovascular disease in the inpatient setting. With respect to research data, we found that prospective ascertainment of clinical data is more often correct when done by research assistants compared with clinicians simultaneously evaluating patients.
authors.affiliation: Department of Emergency Medicine, Hospital of the University of Pennsylvania, Philadelphia, PA, USA.
authors.forename: Carlos O
authors.initials: CO
authors.lastname: Cruz
authors.forename: Emily B
authors.initials: EB
authors.lastname: Meshberg
authors.forename: Frances S
authors.initials: FS
authors.lastname: Shofer
authors.forename: Christine M
authors.initials: CM
authors.lastname: McCusker
authors.forename: Anna Marie
authors.initials: AM
authors.lastname: Chang
authors.forename: Judd E
authors.initials: JE
authors.lastname: Hollander
mesh_terms: D054058:Acute Coronary Syndrome / Q000175:diagnosis*; D003625:Data Collection / Q000379:methods*; D004635:Emergency Medicine / Q000706:statistics & numerical data*; D004636:Emergency Service, Hospital / Q000706:statistics & numerical data*; D005260:Female; D006801:Humans; D008297:Male; D008487:Medical History Taking / Q000706:statistics & numerical data*; D008875:Middle Aged; D010820:Physicians; D011446:Prospective Studies; D015203:Reproducibility of Results; D012108:Research Personnel