title: The reproducibility of assessing radiological reporting: studies from the development of the General Medical Council's Performance Procedures.
abstract: OBJECTIVE
  To investigate the reproducibility of peer ratings of consultant radiologists' reports, as part of the new General Medical Council (GMC) Performance Procedures.
  
  
  METHODS
  An evaluation protocol was piloted, used in a blocked, balanced, randomized generalizability analysis with three blocks of three judges (raters), each rating 30 reports from 10 radiologists, and re-rated to estimate intrarater reliability with conventional statistics (kappa).
  
  
  METHODS
  Rating was performed at the Royal College of Radiologists. Volunteers were sampled from 23 departments of radiology in university teaching and district general hospitals.
  
  
  METHODS
  A nationally drawn non-random sample of 30 consultant radiologists contributing a total of 900 reports. Three trained and six non-trained judges were used in the rating analysis.
  
  
  RESULTS
  A protocol was generated that was usable by judges. Generalizable results would be obtained with not less than three judges all rating the same 60 reports from a radiologist.
  
  
  CONCLUSIONS
  Any assessment of performance of technical abilities in this field will need to use multiple assessors, basing judgements on an adequate sample of reports.
authors.affiliation: Centre for Medical and Health Science Education, Monash University, Melbourne, Victoria, Australia.
authors.forename: B C
authors.initials: BC
authors.lastname: Jolly
authors.forename: B
authors.initials: B
authors.lastname: Ayers
authors.forename: M M
authors.initials: MM
authors.lastname: Macdonald
authors.forename: P
authors.initials: P
authors.lastname: Armstrong
authors.forename: A H
authors.initials: AH
authors.lastname: Chalmers
authors.forename: G
authors.initials: G
authors.lastname: Roberts
authors.forename: L H
authors.initials: LH
authors.lastname: Southgate
mesh_terms: D002983:Clinical Competence / Q000592:standards*; D003657:Decision Making; D003951:Diagnostic Errors / Q000145:classification / Q000706:statistics & numerical data; D006801:Humans; D007600:Judgment; D008485:Medical Audit / Q000145:classification / Q000592:standards*; D018024:Peer Review, Health Care / Q000592:standards*; D011871:Radiology / Q000592:standards*; D015203:Reproducibility of Results; D012955:Societies, Medical; D013222:State Medicine / Q000592:standards; D006113:United Kingdom