Interrater reliability and accuracy of clinicians and trained research assistants performing prospective data collection in emergency department patients with potential acute coronary syndrome.
OBJECTIVE
Clinical research requires high-quality data collection. Data collected at the emergency department evaluation is generally considered more precise than data collected through chart abstraction but is cumbersome and time consuming. We test whether trained research assistants without a medical background can obtain clinical research data as accurately as physicians. We hypothesize that they would be at least as accurate because they would not be distracted by clinical requirements.


METHODS
We conducted a prospective comparative study of 33 trained research assistants and 39 physicians (35 residents) to assess interrater reliability with respect to guideline-recommended clinical research data. Immediately after the research assistant and clinician evaluation, the data were compared by a tiebreaker third person who forced the patient to choose one of the 2 answers as the correct one when responses were discordant. Crude percentage agreement and interrater reliability were assessed (kappa statistic).


RESULTS
One hundred forty-three patients were recruited (mean age 50.7 years; 47% female patients). Overall, the median agreement was 81% (interquartile range [IQR] 73% to 92%) and interrater reliability was fair (kappa value 0.36 [IQR 0.26 to 0.52]) but varied across categories of data: cardiac risk factors (median 86% [IQR 81% to 93%]; median 0.69 [IQR 0.62 to 0.83]), other cardiac history (median 93% [IQR 79% to 95%]; median 0.56 [IQR 0.29 to 0.77]), pain location (median 92% [IR 86% to 94%]; median 0.37 [IQR 0.25 to 0.29]), radiation (median 86% [IQR 85% to 87%]; median 0.37 [IQR 0.26 to 0.42]), quality (median 85% [IQR 75% to 94%]; median 0.29 [IQR 0.23 to 0.40]), and associated symptoms (median 74% [IQR 65% to 78%]; median 0.28 [IQR 0.20 to 0.40]). When discordant information was obtained, the research assistant was more often correct (median 64% [IQR 53% to 72%]).


CONCLUSIONS
The relatively fair interrater reliability observed in our study is consistent with previous studies evaluating interrater reliability for cardiovascular disease in the inpatient setting. With respect to research data, we found that prospective ascertainment of clinical data is more often correct when done by research assistants compared with clinicians simultaneously evaluating patients.
Cruz
Carlos O
CO
Department of Emergency Medicine, Hospital of the University of Pennsylvania, Philadelphia, PA, USA.
Meshberg
Emily B
EB
Shofer
Frances S
FS
McCusker
Christine M
CM
Chang
Anna Marie
AM
Hollander
Judd E
JE
D054058:Acute Coronary Syndrome / Q000175:diagnosis*; D003625:Data Collection / Q000379:methods*; D004635:Emergency Medicine / Q000706:statistics & numerical data*; D004636:Emergency Service, Hospital / Q000706:statistics & numerical data*; D005260:Female; D006801:Humans; D008297:Male; D008487:Medical History Taking / Q000706:statistics & numerical data*; D008875:Middle Aged; D010820:Physicians; D011446:Prospective Studies; D015203:Reproducibility of Results; D012108:Research Personnel