title: Being reliable: issues in determining the reliability and making sense of observations of adults with congenital deafblindness?
abstract: BACKGROUND
  Most research into interactions with people who are congenitally deafblind involves observational data. In order for practitioners and researchers to have confidence in the findings of observational studies, researchers need to demonstrate that the processes employed are replicable and trustworthy. This paper draws on data from an observational study of adults with congenital deafblindness to illustrate issues in determining inter-rater reliability, and interpreting observational data.
  
  
  METHODS
  Data from 34 10-min observations of adults with congenital deafblindness and their interactions with support staff were assessed for inter-rater reliability using percentage agreement calculated in three different ways and Cohen's κ.
  
  
  RESULTS
  Large variation resulted from the different ways in which inter-rater reliability was calculated, largely due to high levels of non-occurrence of many behaviours in the coding tool used.
  
  
  CONCLUSIONS
  This study highlights the need to exercise caution when ascertaining the reliability of observational studies and demonstrates the value in using multiple methods for calculating inter-rater reliability. The paper concludes with an examination of the potential merits of using consensus coding in observational studies of interactions with people with congenital deafblindness or profound intellectual and multiple disabilities.
authors.affiliation: Division of Disability Studies, RMIT University, Melbourne, Vic., Australia. merryprain@gmail.com
authors.forename: M I
authors.initials: MI
authors.lastname: Prain
authors.forename: K R
authors.initials: KR
authors.lastname: McVilly
authors.forename: P
authors.initials: P
authors.lastname: Ramcharan
mesh_terms: D000328:Adult; D054062:Deaf-Blind Disorders / Q000523:psychology / Q000534:rehabilitation*; D005260:Female; D006801:Humans; D008607:Intellectual Disability / Q000523:psychology / Q000534:rehabilitation*; D008297:Male; D015588:Observer Variation; D011594:Psychometrics / Q000592:standards* / Q000706:statistics & numerical data*; D015203:Reproducibility of Results; D012112:Residential Facilities; D012919:Social Behavior; D013192:Staff Development / Q000592:standards / Q000706:statistics & numerical data