title: Reliability of the amsterdam clinical challenge scale (ACCS): a new instrument to assess the level of difficulty of patient cases in medical education.
abstract: BACKGROUND
  In problem-based medical curricula, consideration should be given to the level of difficulty of patient cases used for training and assessment. The Amsterdam Clinical Challenge Scale (ACCS) has been developed to assess the degree of difficulty of patient cases in a systematic and reproducible manner. To determine the reliability of the instrument two research questions were addressed: (1) How many judges are required, on the basis of the total score of the ACCS, to obtain a reliable estimate of the difficulty of a single case? (2) How many cases and/or how many judges are needed to reach an acceptable level of reliability of the total score of the ACCS?
  
  
  METHODS
  Four judges scored 36 patient scripts reflecting a wide range of patient problems encountered in general practice. Each script was scored four times. In the reliability analysis, the generalizability theory was applied.
  
  
  RESULTS
  The results show that the judges did, indeed, use the whole range of difficulty ratings. When the ACCS is applied to a single case, eight or more judges are needed to reach an acceptable level of reliability. When more cases are involved, fewer judges are needed; for 10 or more cases one judge will be sufficient.
  
  
  CONCLUSIONS
  Given the typical length, for example of an objective structured clinical examination, the ACCS makes it possible to provide a reliable estimate of the level of difficulty of such a test with only a limited number of judges.
authors.affiliation: Faculty of Medicine, Vrije Universiteit, Amsterdam, The Netherlands.
authors.forename: A J
authors.initials: AJ
authors.lastname: Gercama
authors.forename: M
authors.initials: M
authors.lastname: de Haan
authors.forename: C P
authors.initials: CP
authors.lastname: van der Vleuten
mesh_terms: D002983:Clinical Competence / Q000592:standards*; D004501:Education, Medical / Q000592:standards*; D006801:Humans; D009426:Netherlands; D016544:Patient Simulation; D010817:Physician-Patient Relations; D018794:Problem-Based Learning / Q000379:methods*; D015203:Reproducibility of Results