title: Inter-rater reliability of the QuIS as an assessment of the quality of staff-inpatient interactions.
abstract: Recent studies of the quality of in-hospital care have used the Quality of Interaction Schedule (QuIS) to rate interactions observed between staff and inpatients in a variety of ward conditions. The QuIS was developed and evaluated in nursing and residential care. We set out to develop methodology for summarising information from inter-rater reliability studies of the QuIS in the acute hospital setting.
  
  
  
  Staff-inpatient interactions were rated by trained staff observing care delivered during two-hour observation periods. Anticipating the possibility of the quality of care varying depending on ward conditions, we selected wards and times of day to reflect the variety of daytime care delivered to patients. We estimated inter-rater reliability using weighted kappa, κw, combined over observation periods to produce an overall, summary estimate, [Formula: see text]. Weighting schemes putting different emphasis on the severity of misclassification between QuIS categories were compared, as were different methods of combining observation period specific estimates.
  
  
  
  Estimated [Formula: see text] did not vary greatly depending on the weighting scheme employed, but we found simple averaging of estimates across observation periods to produce a higher value of inter-rater reliability due to over-weighting observation periods with fewest interactions.
  
  
  
  We recommend that researchers evaluating the inter-rater reliability of the QuIS by observing staff-inpatient interactions during observation periods representing the variety of ward conditions in which care takes place, should summarise inter-rater reliability by κw, weighted according to our scheme A4. Observation period specific estimates should be combined into an overall, single summary statistic [Formula: see text], using a random effects approach, with [Formula: see text], to be interpreted as the mean of the distribution of κwacross the variety of ward conditions. We draw attention to issues in the analysis and interpretation of inter-rater reliability studies incorporating distinct phases of data collection that may generalise more widely.
authors.affiliation: Medical Statistics Group, Faculty of Medicine, Southampton General Hospital, Mailpoint 805, Level B, South Academic Block, Southampton, SO16 6YD, UK.
authors.forename: Ines
authors.initials: I
authors.lastname: Mesa-Eguiagaray
authors.affiliation: Southampton Statistical Sciences Research Institute & Mathematical Sciences, University of Southampton, Southampton, UK.
authors.forename: Dankmar
authors.initials: D
authors.lastname: Böhning
authors.affiliation: Faculty of Health Sciences, University of Southampton, Southampton, UK.
authors.forename: Chris
authors.initials: C
authors.lastname: McLean
authors.affiliation: Faculty of Health Sciences, University of Southampton, Southampton, UK.
authors.forename: Peter
authors.initials: P
authors.lastname: Griffiths
authors.affiliation: Faculty of Health Sciences, University of Southampton, Southampton, UK.
authors.forename: Jackie
authors.initials: J
authors.lastname: Bridges
authors.affiliation: Medical Statistics Group, Faculty of Medicine, Southampton General Hospital, Mailpoint 805, Level B, South Academic Block, Southampton, SO16 6YD, UK. rmp@soton.ac.uk.
authors.forename: Ruth M
authors.initials: RM
authors.lastname: Pickering
mesh_terms: D000465:Algorithms; D003627:Data Interpretation, Statistical; D006801:Humans; D007297:Inpatients; D008503:Medical Staff; D011369:Professional-Patient Relations; D011785:Quality Assurance, Health Care / Q000379:methods*; D015203:Reproducibility of Results
keywords: Averaging; Collapsing; QuIS; Random effects meta-analysis; Weighted kappa