title: Inter-rater agreement in evaluation of disability: systematic review of reproducibility studies.
abstract: OBJECTIVE
  To explore agreement among healthcare professionals assessing eligibility for work disability benefits.
  
  
  METHODS
  Systematic review and narrative synthesis of reproducibility studies.
  
  
  METHODS
  Medline, Embase, and PsycINFO searched up to 16 March 2016, without language restrictions, and review of bibliographies of included studies.
  
  
  METHODS
  Observational studies investigating reproducibility among healthcare professionals performing disability evaluations using a global rating of working capacity and reporting inter-rater reliability by a statistical measure or descriptively. Studies could be conducted in insurance settings, where decisions on ability to work include normative judgments based on legal considerations, or in research settings, where decisions on ability to work disregard normative considerations. : Teams of paired reviewers identified eligible studies, appraised their methodological quality and generalisability, and abstracted results with pretested forms. As heterogeneity of research designs and findings impeded a quantitative analysis, a descriptive synthesis stratified by setting (insurance or research) was performed.
  
  
  RESULTS
  From 4562 references, 101 full text articles were reviewed. Of these, 16 studies conducted in an insurance setting and seven in a research setting, performed in 12 countries, met the inclusion criteria. Studies in the insurance setting were conducted with medical experts assessing claimants who were actual disability claimants or played by actors, hypothetical cases, or short written scenarios. Conditions were mental (n=6, 38%), musculoskeletal (n=4, 25%), or mixed (n=6, 38%). Applicability of findings from studies conducted in an insurance setting to real life evaluations ranged from generalisable (n=7, 44%) and probably generalisable (n=3, 19%) to probably not generalisable (n=6, 37%). Median inter-rater reliability among experts was 0.45 (range intraclass correlation coefficient 0.86 to κ-0.10). Inter-rater reliability was poor in six studies (37%) and excellent in only two (13%). This contrasts with studies conducted in the research setting, where the median inter-rater reliability was 0.76 (range 0.91-0.53), and 71% (5/7) studies achieved excellent inter-rater reliability. Reliability between assessing professionals was higher when the evaluation was guided by a standardised instrument (23 studies, P=0.006). No such association was detected for subjective or chronic health conditions or the studies' generalisability to real world evaluation of disability (P=0.46, 0.45, and 0.65, respectively).
  
  
  CONCLUSIONS
  Despite their common use and far reaching consequences for workers claiming disabling injury or illness, research on the reliability of medical evaluations of disability for work is limited and indicates high variation in judgments among assessing professionals. Standardising the evaluation process could improve reliability. Development and testing of instruments and structured approaches to improve reliability in evaluation of disability are urgently needed.
authors.affiliation: Evidence-based Insurance Medicine (EbIM), Research and Education, Department Clinical Research, University Basel Hospital, University of Basel, Spitalstrasse 8 + 12, CH-4031 Basel, Switzerland.|Institute for Complementary and Integrative Medicine, University Hospital Zurich and University of Zurich, Zurich, Switzerland.
authors.forename: Jürgen
authors.initials: J
authors.lastname: Barth
authors.affiliation: Evidence-based Insurance Medicine (EbIM), Research and Education, Department Clinical Research, University Basel Hospital, University of Basel, Spitalstrasse 8 + 12, CH-4031 Basel, Switzerland.
authors.forename: Wout E L
authors.initials: WE
authors.lastname: de Boer
authors.affiliation: Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, ON L8S 4K1, Canada.|Department of Anaesthesia, McMaster University, Hamilton, ON L8S 4K1, Canada.|The Michael G DeGroote Institute for Pain Research and Care, McMaster University, Hamilton, ON L8S 4K1, Canada.
authors.forename: Jason W
authors.initials: JW
authors.lastname: Busse
authors.affiliation: Coronel Institute of Occupational Health, Academic Medical Centre, University of Amsterdam, Amsterdam, Netherlands.|Research Centre for Insurance Medicine, AMC-UMCG-UWV-VUmc, Amsterdam, Netherlands.
authors.forename: Jan L
authors.initials: JL
authors.lastname: Hoving
authors.affiliation: Evidence-based Insurance Medicine (EbIM), Research and Education, Department Clinical Research, University Basel Hospital, University of Basel, Spitalstrasse 8 + 12, CH-4031 Basel, Switzerland.
authors.forename: Sarah
authors.initials: S
authors.lastname: Kedzia
authors.affiliation: Department of Anaesthesia, McMaster University, Hamilton, ON L8S 4K1, Canada.
authors.forename: Rachel
authors.initials: R
authors.lastname: Couban
authors.affiliation: School of Applied Psychology, Institute Humans in Complex Systems, Olten, Switzerland.
authors.forename: Katrin
authors.initials: K
authors.lastname: Fischer
authors.affiliation: Evidence-based Insurance Medicine (EbIM), Research and Education, Department Clinical Research, University Basel Hospital, University of Basel, Spitalstrasse 8 + 12, CH-4031 Basel, Switzerland.
authors.forename: David Y
authors.initials: DY
authors.lastname: von Allmen
authors.affiliation: Dutch National Institute for Employee Benefits Schemes, Groningen, Netherlands.|Department of Health Sciences, Community and Occupational Medicine, University Medical Centre Groningen, Netherlands.
authors.forename: Jerry
authors.initials: J
authors.lastname: Spanjer
authors.affiliation: Evidence-based Insurance Medicine (EbIM), Research and Education, Department Clinical Research, University Basel Hospital, University of Basel, Spitalstrasse 8 + 12, CH-4031 Basel, Switzerland regina.kunz@usb.ch.
authors.forename: Regina
authors.initials: R
authors.lastname: Kunz
mesh_terms: D004185:Disability Evaluation; D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results