title: Ambiguities and conflicting results: the limitations of the kappa statistic in establishing the interrater reliability of the Irish nursing minimum data set for mental health: a discussion paper.
abstract: BACKGROUND
  In a study to establish the interrater reliability of the Irish Nursing Minimum Data Set (I-NMDS) for mental health difficulties relating to the choice of reliability test statistic were encountered.
  
  
  OBJECTIVE
  The objective of this paper is to highlight the difficulties associated with testing interrater reliability for an ordinal scale using a relatively homogenous sample and the recommended kw statistic.
  
  
  METHODS
  One pair of mental health nurses completed the I-NMDS for mental health for a total of 30 clients attending a mental health day centre over a two-week period. Data was analysed using the kw and percentage agreement statistics.
  
  
  RESULTS
  A total of 34 of the 38 I-NMDS for mental health variables with lower than acceptable levels of kw reliability scores achieved acceptable levels of reliability according to their percentage agreement scores.
  
  
  CONCLUSIONS
  The study findings implied that, due to the homogeneity of the sample, low variability within the data resulted in the 'base rate problem' associated with the use of kw statistic. Conclusions point to the interpretation of kw in tandem with percentage agreement scores. Suggestions that kw scores were low due to chance agreement and that one should strive to use a study sample with known variability are queried.
authors.affiliation: Dublin City University, Ollscoil Chathair Bhaile Atha Cliath 9, Eire, Dublin, Ireland. roisin.morris@dcu.ie
authors.forename: Roisin
authors.initials: R
authors.lastname: Morris
authors.forename: Padraig
authors.initials: P
authors.lastname: MacNeela
authors.forename: Anne
authors.initials: A
authors.lastname: Scott
authors.forename: Pearl
authors.initials: P
authors.lastname: Treacy
authors.forename: Abbey
authors.initials: A
authors.lastname: Hyde
authors.forename: Julian
authors.initials: J
authors.lastname: O'Brien
authors.forename: Daniella
authors.initials: D
authors.lastname: Lehwaldt
authors.forename: Anne
authors.initials: A
authors.lastname: Byrne
authors.forename: Jonathan
authors.initials: J
authors.lastname: Drennan
mesh_terms: D003625:Data Collection / Q000379:methods / Q000592:standards; D003627:Data Interpretation, Statistical; D006801:Humans; D007494:Ireland; D009730:Nursing Assessment / Q000592:standards; D015403:Nursing Evaluation Research / Q000379:methods* / Q000592:standards; D015588:Observer Variation; D010343:Patient Admission; D011568:Psychiatric Nursing / Q000379:methods; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity