Improved inter-observer agreement of an expert review panel in an oncology treatment trial--Insights from a structured interventional process.
OBJECTIVE
Oncologic imaging is a key for successful cancer treatment. While the quality assurance (QA) of image acquisition protocols has already been focussed, QA of reading and reporting offers still room for improvement. The latter was addressed in the context of a prospective multicentre trial on fluoro-deoxyglucose (FDG)-positron-emission tomography (PET)/CT-based chemoradiotherapy for locally advanced non-small cell lung cancer (NSCLC).


METHODS
An expert panel was prospectively installed performing blinded reviews of mediastinal NSCLC involvement in FDG-PET/CT. Due to a high initial reporting inter-observer disagreement, the independent data monitoring committee (IDMC) triggered an interventional harmonisation process, which overall involved 11 experts uttering 6855 blinded diagnostic statements. After assessing the baseline inter-observer agreement (IOA) of a blinded re-review (phase 1), a discussion process led to improved reading criteria (phase 2). Those underwent a validation study (phase 3) and were then implemented into the study routine. After 2 months (phase 4) and 1 year (phase 5), the IOA was reassessed.


RESULTS
The initial overall IOA was moderate (kappa 0.52 CT; 0.53 PET). After improvement of reading criteria, the kappa values improved substantially (kappa 0.61 CT; 0.66 PET), which was retained until the late reassessment (kappa 0.71 CT; 0.67 PET). Subjective uncertainty was highly predictive for low IOA.


CONCLUSIONS
The IOA of an expert panel was significantly improved by a structured interventional harmonisation process which could be a model for future clinical trials. Furthermore, the low IOA in reporting nodal involvement in NSCLC may bear consequences for individual patient care.
Department of Radiation Oncology, Freiburg University Medical Center, Germany; German Cancer Consortium (DKTK), Heidelberg, Partner Site Freiburg, Germany. Electronic address: ursula.nestle@uniklinik-freiburg.de.
Ursula
U
Nestle
Department of Radiation Oncology, Freiburg University Medical Center, Germany; Department of Nuclear Medicine, Freiburg University Medical Center, Germany.
Hans Christian
HC
Rischke
Department of Nuclear Medicine, Marienhospital Stuttgart, Germany.
Susanne Martina
SM
Eschmann
Department of Nuclear Medicine, Klinikum Potsdam, Germany.
Gabriele
G
Holl
Department of Nuclear Medicine, HELIOS Hospital Wuppertal, Germany; Faculty of Health and Department of Medicine, University of Witten/Herdecke, Germany.
Marco
M
Tosch
Department of Nuclear Medicine, University Medical Center Mainz, Germany.
Matthias
M
Miederer
Vivantes Institute für Nuclear Medicine Mitte/Nord, Berlin, Germany.
Michail
M
Plotkin
Department of Nuclear Medicine, University of Bonn, Germany.
Markus
M
Essler
Department of Nuclear Medicine, Municipal Hospital Karlsruhe, Germany.
Cornelia
C
Puskas
Department of Radiation Oncology, Freiburg University Medical Center, Germany.
Tanja
T
Schimek-Jasch
Department of Radiation Oncology, Freiburg University Medical Center, Germany; Department Radiation-Onkology, Ortenau Klinikum Offenburg-Gengenbach, Germany.
Viola
V
Duncker-Rohr
Department of Radiation Oncology, Freiburg University Medical Center, Germany.
Friederike
F
Rühl
Department of Radiation Oncology, Freiburg University Medical Center, Germany.
Anja
A
Leifert
Department of Nuclear Medicine, Freiburg University Medical Center, Germany.
Michael
M
Mix
Department of Radiation Oncology, Freiburg University Medical Center, Germany; German Cancer Consortium (DKTK), Heidelberg, Partner Site Freiburg, Germany.
Anca-Ligia
AL
Grosu
Institute of Medical Biostatistics, Epidemiology and Informatics, University Medical Center, Johannes Gutenberg University Mainz, Germany.
Jochem
J
König
Center of Medical Biometry and Medical Informatics, Freiburg University Medical Center, Germany.
Werner
W
Vach
D002289:Carcinoma, Non-Small-Cell Lung / Q000175:diagnosis* / Q000628:therapy; D059248:Chemoradiotherapy / Q000379:methods; D004311:Double-Blind Method; D019788:Fluorodeoxyglucose F18; D006801:Humans; D008175:Lung Neoplasms / Q000175:diagnosis* / Q000628:therapy; D015588:Observer Variation; D017063:Outcome Assessment, Health Care / Q000379:methods; D049268:Positron-Emission Tomography / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D014057:Tomography, X-Ray Computed / Q000379:methods*
D019788:Fluorodeoxyglucose F18
Clinical trial; Expert review; FDG; Lung cancer; PET/CT; Radiotherapy; Reading criteria