title: Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) checklist.
abstract: BACKGROUND
  The COSMIN checklist is a tool for evaluating the methodological quality of studies on measurement properties of health-related patient-reported outcomes. The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist (n = 114).
  
  
  METHODS
  75 articles evaluating measurement properties were randomly selected from the bibliographic database compiled by the Patient-Reported Outcome Measurement Group, Oxford, UK. Raters were asked to assess the methodological quality of three articles, using the COSMIN checklist. In a one-way design, percentage agreement and intraclass kappa coefficients or quadratic-weighted kappa coefficients were calculated for each item.
  
  
  RESULTS
  88 raters participated. Of the 75 selected articles, 26 articles were rated by four to six participants, and 49 by two or three participants. Overall, percentage agreement was appropriate (68% was above 80% agreement), and the kappa coefficients for the COSMIN items were low (61% was below 0.40, 6% was above 0.75). Reasons for low inter-rater agreement were need for subjective judgement, and accustom to different standards, terminology and definitions.
  
  
  CONCLUSIONS
  Results indicated that raters often choose the same response option, but that it is difficult on item level to distinguish between articles. When using the COSMIN checklist in a systematic review, we recommend getting some training and experience, completing it by two independent raters, and reaching consensus on one final rating. Instructions for using the checklist are improved.
authors.affiliation: Department of Epidemiology and Biostatistics and the EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands. w.mokkink@vumc.nl
authors.forename: Lidwine B
authors.initials: LB
authors.lastname: Mokkink
authors.forename: Caroline B
authors.initials: CB
authors.lastname: Terwee
authors.forename: Elizabeth
authors.initials: E
authors.lastname: Gibbons
authors.forename: Paul W
authors.initials: PW
authors.lastname: Stratford
authors.forename: Jordi
authors.initials: J
authors.lastname: Alonso
authors.forename: Donald L
authors.initials: DL
authors.lastname: Patrick
authors.forename: Dirk L
authors.initials: DL
authors.lastname: Knol
authors.forename: Lex M
authors.initials: LM
authors.lastname: Bouter
authors.forename: Henrica C W
authors.initials: HC
authors.lastname: de Vet
mesh_terms: D057189:Checklist / Q000379:methods; D003627:Data Interpretation, Statistical; D006305:Health Status Indicators; D015588:Observer Variation; D017063:Outcome Assessment, Health Care / Q000592:standards*; D010506:Periodicals as Topic / Q000592:standards*; D036301:Qualitative Research; D015203:Reproducibility of Results