title: Assessment and improvement of radiation oncology trainee contouring ability utilizing consensus-based penalty metrics.
abstract: BACKGROUND
  The objective of this study was to develop and assess the feasibility of utilizing consensus-based penalty metrics for the purpose of critical structure and organ at risk (OAR) contouring quality assurance and improvement.
  
  
  METHODS
  A Delphi study was conducted to obtain consensus on contouring penalty metrics to assess trainee-generated OAR contours. Voxel-based penalty metric equations were used to score regions of discordance between trainee and expert contour sets. The utility of these penalty metric scores for objective feedback on contouring quality was assessed by using cases prepared for weekly radiation oncology radiation oncology trainee treatment planning rounds.
  
  
  RESULTS
  In two Delphi rounds, six radiation oncology specialists reached agreement on clinical importance/impact and organ radiosensitivity as the two primary criteria for the creation of the Critical Structure Inter-comparison of Segmentation (CriSIS) penalty functions. Linear/quadratic penalty scoring functions (for over- and under-contouring) with one of four levels of severity (none, low, moderate and high) were assigned for each of 20 OARs in order to generate a CriSIS score when new OAR contours are compared with reference/expert standards. Six cases (central nervous system, head and neck, gastrointestinal, genitourinary, gynaecological and thoracic) then were used to validate 18 OAR metrics through comparison of trainee and expert contour sets using the consensus derived CriSIS functions. For 14 OARs, there was an improvement in CriSIS score post-educational intervention.
  
  
  CONCLUSIONS
  The use of consensus-based contouring penalty metrics to provide quantitative information for contouring improvement is feasible.
authors.affiliation: Department of Oncology, University of Western Ontario and London Health Sciences Centre, London, Ontario, Canada.
authors.forename: Abhirami
authors.initials: A
authors.lastname: Hallock
authors.forename: Glenn
authors.initials: G
authors.lastname: Bauman
authors.forename: Nancy
authors.initials: N
authors.lastname: Read
authors.forename: David
authors.initials: D
authors.lastname: D'Souza
authors.forename: Francisco
authors.initials: F
authors.lastname: Perera
authors.forename: Inge
authors.initials: I
authors.lastname: Aivas
authors.forename: Lara
authors.initials: L
authors.lastname: Best
authors.forename: Jeffrey
authors.initials: J
authors.lastname: Cao
authors.forename: Alexander V
authors.initials: AV
authors.lastname: Louie
authors.forename: Ericka
authors.initials: E
authors.lastname: Wiebe
authors.forename: Tracy
authors.initials: T
authors.lastname: Sexton
authors.forename: Stewart
authors.initials: S
authors.lastname: Gaede
authors.forename: Jerry
authors.initials: J
authors.lastname: Battista
authors.forename: George
authors.initials: G
authors.lastname: Rodrigues
mesh_terms: D004521:Educational Measurement / Q000379:methods / Q000592:standards*; D009864:Ontario; D018787:Radiation Oncology / Q000193:education* / Q000592:standards*; D061089:Radiotherapy, Image-Guided / Q000592:standards*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D014057:Tomography, X-Ray Computed / Q000379:methods*