Methods to achieve high interrater reliability in data collection from primary care medical records.
OBJECTIVE
We assessed interrater reliability (IRR) of chart abstractors within a randomized trial of cardiovascular care in primary care. We report our findings, and outline issues and provide recommendations related to determining sample size, frequency of verification, and minimum thresholds for 2 measures of IRR: the κ statistic and percent agreement.


METHODS
We designed a data quality monitoring procedure having 4 parts: use of standardized protocols and forms, extensive training, continuous monitoring of IRR, and a quality improvement feedback mechanism. Four abstractors checked a 5% sample of charts at 3 time points for a predefined set of indicators of the quality of care. We set our quality threshold for IRR at a κ of 0.75, a percent agreement of 95%, or both.


RESULTS
Abstractors reabstracted a sample of charts in 16 of 27 primary care practices, checking a total of 132 charts with 38 indicators per chart. The overall κ across all items was 0.91 (95% confidence interval, 0.90-0.92) and the overall percent agreement was 94.3%, signifying excellent agreement between abstractors. We gave feedback to the abstractors to highlight items that had a κ of less than 0.70 or a percent agreement less than 95%. No practice had to have its charts abstracted again because of poor quality.


CONCLUSIONS
A 5% sampling of charts for quality control using IRR analysis yielded κ and agreement levels that met or exceeded our quality thresholds. Using 3 time points during the chart audit phase allows for early quality control as well as ongoing quality monitoring. Our results can be used as a guide and benchmark for other medical chart review studies in primary care.
Department of Family Medicine, University of Ottawa, Ottawa, Ontario, Canada. cliddy@bruyere.org
Clare
C
Liddy
Miriam
M
Wiens
William
W
Hogg
D000043:Abstracting and Indexing / Q000592:standards* / Q000706:statistics & numerical data*; D003625:Data Collection / Q000592:standards*; D006801:Humans; D008499:Medical Records; D015588:Observer Variation; D011320:Primary Health Care; D011786:Quality Control; D015203:Reproducibility of Results; D018401:Sample Size
J Clin Epidemiol. 1990;43(6):543-9
2348207
Am J Epidemiol. 2003 Mar 15;157(6):546-51
12631545
Ann Emerg Med. 2005 Apr;45(4):448-51
15795729
Ann Emerg Med. 2005 Apr;45(4):452-5
15795730
Am J Epidemiol. 2005 May 15;161(10):974-7
15870162
Can Fam Physician. 2010 May;56(5):495-6
20463278
Acad Emerg Med. 2004 Feb;11(2):187-92
14759964
Appl Nurs Res. 2005 Feb;18(1):50-4
15812736
Eval Health Prof. 2009 Sep;32(3):281-98
19679636
Stat Med. 1992 Aug;11(11):1511-9
1410963
J Clin Epidemiol. 1993 May;46(5):423-9
8501467
Ann Emerg Med. 1996 Mar;27(3):305-8
8599488
Med Care. 1998 Jun;36(6):851-67
9630127
Air Med J. 2007 Sep-Oct;26(5):206-10
17765825
Am J Manag Care. 2002 Sep;8(9):787-93
12234019
Eval Health Prof. 1997 Jun;20(2):146-63
10183318
Jt Comm J Qual Improv. 2000 Mar;26(3):115-36
10709146
BMC Med Res Methodol. 2008 May 09;8:29
18471298
Acad Emerg Med. 2005 Sep;12(9):884-95
16141025
Phys Ther. 2005 Mar;85(3):257-68
15733050
BMC Med Res Methodol. 2006 Nov 08;6:54
17092344
J Clin Epidemiol. 2008 Apr;61(4):394-401
18313565
J Can Acad Child Adolesc Psychiatry. 2006 Aug;15(3):126-34
18392182
J Clin Epidemiol. 2005 Sep;58(9):918-23
16085195
Health Aff (Millwood). 2006 Nov-Dec;25(6):w555-71
17102164