title: Sample size requirements for training to a kappa agreement criterion on clinical dementia ratings.
abstract: The Clinical Dementia Rating (CDR) is a valid and reliable global measure of dementia severity. Diagnosis and transition across stages hinge on its consistent administration. Reports of CDR ratings reliability have been based on 1 or 2 test cases at each severity level; agreement (kappa) statistics based on so few rated cases have large error, and confidence intervals are incorrect. Simulations varied the number of test cases, and their distribution across CDR stage; to derive the sample size yielding a 95% confidence that estimated is at least 0.60. We found that testing raters on 5 or more patients per CDR level (total N=25) will yield the desired confidence in estimated kappa, and if the test involves greater representation of CDR stages that are harder to evaluate, at least 42 ratings are needed. Testing newly trained raters with at least 5 patients per CDR stage will provide valid estimation of rater consistency, given the point estimate for kappa is roughly 0.80; fewer test cases increases the standard error and unequal distribution of test cases across CDR stages will lower kappa and increase error.
authors.affiliation: Department of Neurology, Georgetown University School of Medicine, Washington, DC 20057, USA. ret7@georgetown.edu
authors.forename: Rochelle E
authors.initials: RE
authors.lastname: Tractenberg
authors.forename: Futoshi
authors.initials: F
authors.lastname: Yumoto
authors.forename: Shelia
authors.initials: S
authors.lastname: Jin
authors.forename: John C
authors.initials: JC
authors.lastname: Morris
mesh_terms: D002986:Clinical Trials as Topic; D003198:Computer Simulation / Q000706:statistics & numerical data; D003704:Dementia / Q000523:psychology; D006801:Humans; D011569:Psychiatric Status Rating Scales / Q000592:standards* / Q000706:statistics & numerical data*; D015203:Reproducibility of Results
references.citation: Biometrics. 2001 Jun;57(2):584-8
references.pmid: 11414588
references.citation: Alzheimer Dis Assoc Disord. 2001 Jul-Sep;15(3):155-61
references.pmid: 11522933
references.citation: Alzheimer Dis Assoc Disord. 2004 Oct-Dec;18(4):219-22
references.pmid: 15592134
references.citation: Br J Psychiatry. 1982 Jun;140:566-72
references.pmid: 7104545
references.citation: Stat Med. 1998 Jan 15;17(1):101-10
references.pmid: 9463853
references.citation: Stat Med. 1992 Aug;11(11):1511-9
references.pmid: 1410963
references.citation: Stat Med. 2006 May 15;25(9):1547-60
references.pmid: 16143966
references.citation: Stat Med. 1998 May 30;17(10):1157-68
references.pmid: 9618775
references.citation: Arch Neurol. 1989 Nov;46(11):1210-1
references.pmid: 2818255
references.citation: Arch Neurol. 1988 Jan;45(1):31-2
references.pmid: 3337672
references.citation: Stat Med. 2000 Mar 15;19(5):723-41
references.pmid: 10700742
references.citation: Neurology. 1993 Nov;43(11):2412-4
references.pmid: 8232972
references.citation: Biometrics. 1977 Mar;33(1):159-74
references.pmid: 843571