title: The Work-ability Support Scale: evaluation of scoring accuracy and rater reliability.
abstract: OBJECTIVE
  The Work-ability Support Scale (WSS) is a new tool designed to assess vocational ability and support needs following onset of acquired disability, to assist decision-making in vocational rehabilitation. In this article, we report an iterative process of development through evaluation of inter- and intra-rater reliability and scoring accuracy, using vignettes. The impact of different methodological approaches to analysis of reliability is highlighted.
  
  
  METHODS
  Following preliminary evaluation using case-histories, six occupational therapists scored vignettes, first individually and then together in two teams. Scoring was repeated blind after 1 month. Scoring accuracy was tested against agreed 'reference standard' vignette scores using intraclass correlation coefficients (ICCs) for total scores and linear-weighted kappas (kw) for individual items. Item-by-item inter- and intra-rater reliability was evaluated for both individual and team scores, using two different statistical methods.
  
  
  RESULTS
  ICCs for scoring accuracy ranged from 0.95 (95 % CI 0.78-0.98) to 0.96 (0.89-0.99) for Part A, and from 0.78 (95 % CI 0.67-0.85) to 0.84 (0.69-0.92) for Part B. Item by item analysis of scoring accuracy, inter- and intra-rater reliability all showed 'substantial' to 'almost perfect' agreement (kw ≥ 0.60) for all Part-A and 8/12 Part-B items, although multi-rater kappa (Fleiss) produced more conservative results (mK = 0.34-0.79). Team rating produced marginal improvements for Part-A but not Part-B. Four problematic contextual items were identified, leading to adjustment of the scoring manual.
  
  
  CONCLUSIONS
  This vignette-based study demonstrates generally acceptable levels of scoring accuracy and reliability for the WSS. Further testing in real-life situations is now warranted.
authors: Department of Palliative Care, Policy and Rehabilitation, School of Medicine, King's College London, London, UK, Lynne.turner-stokes@dial.pipex.com.
authors: Lynne
authors: L
authors: Turner-Stokes
authors: Joanna
authors: J
authors: Fadyl
authors: Hilary
authors: H
authors: Rose
authors: Heather
authors: H
authors: Williams
authors: Philip
authors: P
authors: Schlüter
authors: Kathryn
authors: K
authors: McPherson
mesh_terms: D004185:Disability Evaluation; D006801:Humans; D020380:Needs Assessment; D011594:Psychometrics; D012050:Rehabilitation, Vocational; D015203:Reproducibility of Results; D011795:Surveys and Questionnaires; D014938:Work Capacity Evaluation