title: Observer variation in target volume delineation of lung cancer related to radiation oncologist-computer interaction: a 'Big Brother' evaluation.
abstract: OBJECTIVE
  To evaluate the process of target volume delineation in lung cancer for optimization of imaging, delineation protocol and delineation software.
  
  
  METHODS
  Eleven radiation oncologists (observers) from five different institutions delineated the Gross Tumor Volume (GTV) including positive lymph nodes of 22 lung cancer patients (stages I-IIIB) on CT only. All radiation oncologist-computer interactions were recorded with a tool called 'Big Brother'. For each radiation oncologist and patient the following issues were analyzed: delineation time, number of delineated points and corrections, zoom levels, level and window (L/W) settings, CT slice changes, use of side windows (coronal and sagittal) and software button use.
  
  
  RESULTS
  The mean delineation time per GTV was 16 min (SD 10 min). The mean delineation time for lymph node positive patients was on average 3 min larger (P = 0.02) than for lymph node negative patients. Many corrections (55%) were due to L/W change (e.g. delineating in mediastinum L/W and then correcting in lung L/W). For the lymph node region, a relatively large number of corrections was found (3.7 corr/cm2), indicating that it was difficult to delineate lymph nodes. For the tumor-atelectasis region, a relative small number of corrections was found (1.0 corr/cm2), indicating that including or excluding atelectasis into the GTV was a clinical decision. Inappropriate use of L/W settings was frequently found (e.g. 46% of all delineated points in the tumor-lung region were delineated in mediastinum L/W settings). Despite a large observer variation in cranial and caudal direction of 0.72 cm (1 SD), the coronal and sagittal side windows were not used in 45 and 60% of the cases, respectively. For the more difficult cases, observer variation was smaller when the coronal and sagittal side windows were used.
  
  
  CONCLUSIONS
  With the 'Big Brother' tool a method was developed to trace the delineation process. The differences between observers concerning the delineation style were large. This study led to recommendations on how to improve delineation accuracy by adapting the delineation protocol (guidelines for L/W use) and delineation software (double window with lung and mediastinum L/W settings at the same time, enforced use of coronal and sagittal views) and including FDG-PET information (lymph nodes and atelectasis).
authors.affiliation: The Netherlands Cancer Institute, Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands.
authors.forename: Roel J H M
authors.initials: RJ
authors.lastname: Steenbakkers
authors.forename: Joop C
authors.initials: JC
authors.lastname: Duppen
authors.forename: Isabelle
authors.initials: I
authors.lastname: Fitton
authors.forename: Kirsten E I
authors.initials: KE
authors.lastname: Deurloo
authors.forename: Lambert
authors.initials: L
authors.lastname: Zijp
authors.forename: Apollonia L J
authors.initials: AL
authors.lastname: Uitterhoeve
authors.forename: Patrick T R
authors.initials: PT
authors.lastname: Rodrigus
authors.forename: Gijsbert W P
authors.initials: GW
authors.lastname: Kramer
authors.forename: Johan
authors.initials: J
authors.lastname: Bussink
authors.forename: Katrien
authors.initials: K
authors.lastname: De Jaeger
authors.forename: José S A
authors.initials: JS
authors.lastname: Belderbos
authors.forename: Augustinus A M
authors.initials: AA
authors.lastname: Hart
authors.forename: Peter J C M
authors.initials: PJ
authors.lastname: Nowak
authors.forename: Marcel
authors.initials: M
authors.lastname: van Herk
authors.forename: Coen R N
authors.initials: CR
authors.lastname: Rasch
mesh_terms: D000368:Aged; D000369:Aged, 80 and over; D002289:Carcinoma, Non-Small-Cell Lung / Q000473:pathology / Q000532:radiotherapy*; D015331:Cohort Studies; D004307:Dose-Response Relationship, Radiation; D004867:Equipment Design; D004869:Equipment Safety; D005069:Evaluation Studies as Topic; D005260:Female; D006801:Humans; D008175:Lung Neoplasms / Q000473:pathology / Q000532:radiotherapy*; D008297:Male; D008875:Middle Aged; D009367:Neoplasm Staging; D015588:Observer Variation; D010818:Practice Patterns, Physicians'; D018787:Radiation Oncology / Q000592:standards / Q000639:trends; D011879:Radiotherapy Dosage; D011880:Radiotherapy Planning, Computer-Assisted / Q000295:instrumentation* / Q000379:methods; D018570:Risk Assessment; D012680:Sensitivity and Specificity; D014057:Tomography, X-Ray Computed; D016896:Treatment Outcome