Ambiguities and conflicting results: the limitations of the kappa statistic in establishing the interrater reliability of the Irish nursing minimum data set for mental health: a discussion paper.
BACKGROUND
In a study to establish the interrater reliability of the Irish Nursing Minimum Data Set (I-NMDS) for mental health difficulties relating to the choice of reliability test statistic were encountered.


OBJECTIVE
The objective of this paper is to highlight the difficulties associated with testing interrater reliability for an ordinal scale using a relatively homogenous sample and the recommended kw statistic.


METHODS
One pair of mental health nurses completed the I-NMDS for mental health for a total of 30 clients attending a mental health day centre over a two-week period. Data was analysed using the kw and percentage agreement statistics.


RESULTS
A total of 34 of the 38 I-NMDS for mental health variables with lower than acceptable levels of kw reliability scores achieved acceptable levels of reliability according to their percentage agreement scores.


CONCLUSIONS
The study findings implied that, due to the homogeneity of the sample, low variability within the data resulted in the 'base rate problem' associated with the use of kw statistic. Conclusions point to the interpretation of kw in tandem with percentage agreement scores. Suggestions that kw scores were low due to chance agreement and that one should strive to use a study sample with known variability are queried.
Morris
Roisin
R
Dublin City University, Ollscoil Chathair Bhaile Atha Cliath 9, Eire, Dublin, Ireland. roisin.morris@dcu.ie
MacNeela
Padraig
P
Scott
Anne
A
Treacy
Pearl
P
Hyde
Abbey
A
O'Brien
Julian
J
Lehwaldt
Daniella
D
Byrne
Anne
A
Drennan
Jonathan
J
D003625:Data Collection / Q000379:methods / Q000592:standards; D003627:Data Interpretation, Statistical; D006801:Humans; D007494:Ireland; D009730:Nursing Assessment / Q000592:standards; D015403:Nursing Evaluation Research / Q000379:methods* / Q000592:standards; D015588:Observer Variation; D010343:Patient Admission; D011568:Psychiatric Nursing / Q000379:methods; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity