Sample size requirements for training to a kappa agreement criterion on clinical dementia ratings.
The Clinical Dementia Rating (CDR) is a valid and reliable global measure of dementia severity. Diagnosis and transition across stages hinge on its consistent administration. Reports of CDR ratings reliability have been based on 1 or 2 test cases at each severity level; agreement (kappa) statistics based on so few rated cases have large error, and confidence intervals are incorrect. Simulations varied the number of test cases, and their distribution across CDR stage; to derive the sample size yielding a 95% confidence that estimated is at least 0.60. We found that testing raters on 5 or more patients per CDR level (total N=25) will yield the desired confidence in estimated kappa, and if the test involves greater representation of CDR stages that are harder to evaluate, at least 42 ratings are needed. Testing newly trained raters with at least 5 patients per CDR stage will provide valid estimation of rater consistency, given the point estimate for kappa is roughly 0.80; fewer test cases increases the standard error and unequal distribution of test cases across CDR stages will lower kappa and increase error.
Tractenberg
Rochelle E
RE
Department of Neurology, Georgetown University School of Medicine, Washington, DC 20057, USA. ret7@georgetown.edu
Yumoto
Futoshi
F
Jin
Shelia
S
Morris
John C
JC
D002986:Clinical Trials as Topic; D003198:Computer Simulation / Q000706:statistics & numerical data; D003704:Dementia / Q000523:psychology; D006801:Humans; D011569:Psychiatric Status Rating Scales / Q000592:standards* / Q000706:statistics & numerical data*; D015203:Reproducibility of Results
Biometrics. 2001 Jun;57(2):584-8
11414588
Alzheimer Dis Assoc Disord. 2001 Jul-Sep;15(3):155-61
11522933
Alzheimer Dis Assoc Disord. 2004 Oct-Dec;18(4):219-22
15592134
Br J Psychiatry. 1982 Jun;140:566-72
7104545
Stat Med. 1998 Jan 15;17(1):101-10
9463853
Stat Med. 1992 Aug;11(11):1511-9
1410963
Stat Med. 2006 May 15;25(9):1547-60
16143966
Stat Med. 1998 May 30;17(10):1157-68
9618775
Arch Neurol. 1989 Nov;46(11):1210-1
2818255
Arch Neurol. 1988 Jan;45(1):31-2
3337672
Stat Med. 2000 Mar 15;19(5):723-41
10700742
Neurology. 1993 Nov;43(11):2412-4
8232972
Biometrics. 1977 Mar;33(1):159-74
843571