title: Experimental studies to improve the reliability and validity of regulatory judgments on health care in the Netherlands: a randomized controlled trial and before and after case study.
abstract: OBJECTIVE
  We examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting.
  
  
  METHODS
  We adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions.
  
  
  RESULTS
  The consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Validity is assessed by examining the relation between the assigned judgments and the corporate standard and expressed by a correlation coefficient. This coefficient was highest after the consensus meeting (0.48). Adjustment of the instrument did not increase reliability and validity coefficients.
  
  
  CONCLUSIONS
  Participating in a consensus meeting improved reliability and validity. Increasing the number of inspectors resulted in both higher reliability and validity values. Organizing consensus meetings and increasing the number of inspectors per regulatory visit seem to be valuable interventions for improving regulatory judgments.
authors.affiliation: O&I, IGZ, Utrecht, The Netherlands.
authors.forename: Saskia M
authors.initials: SM
authors.lastname: Tuijn
authors.forename: Huub
authors.initials: H
authors.lastname: van den Bergh
authors.forename: Paul
authors.initials: P
authors.lastname: Robben
authors.forename: Frans
authors.initials: F
authors.lastname: Janssens
mesh_terms: D057189:Checklist / Q000592:standards*; D032921:Consensus; D003245:Consensus Statements as Topic; D033161:Government Regulation; D006801:Humans; D007600:Judgment; D009426:Netherlands; D009735:Nursing Homes / Q000331:legislation & jurisprudence*; D019982:Organizational Case Studies; D003668:Pressure Ulcer / Q000628:therapy; D058996:Quality Improvement; D012306:Risk
keywords: evaluation; health services research