Inter-rater reliability of surgical reviews for AREN03B2: a COG renal tumor committee study.
OBJECTIVE
The Children's Oncology Group (COG) renal tumor study (AREN03B2) requires real-time central review of radiology, pathology, and the surgical procedure to determine appropriate risk-based therapy. The purpose of this study was to determine the inter-rater reliability of the surgical reviews.


METHODS
Of the first 3200 enrolled AREN03B2 patients, a sample of 100 enriched for blood vessel involvement, spill, rupture, and lymph node involvement was selected for analysis. The surgical assessment was then performed independently by two blinded surgical reviewers and compared to the original assessment, which had been completed by another of the committee surgeons. Variables assessed included surgeon-determined local tumor stage, overall disease stage, type of renal procedure performed, presence of tumor rupture, occurrence of intraoperative tumor spill, blood vessel involvement, presence of peritoneal implants, and interpretation of residual disease. Inter-rater reliability was measured using the Fleiss' Kappa statistic two-sided hypothesis tests (Kappa, p-value).


RESULTS
Local tumor stage correlated in all 3 reviews except in one case (Kappa=0.9775, p<0.001). Similarly, overall disease stage had excellent correlation (0.9422, p<0.001). There was strong correlation for type of renal procedure (0.8357, p<0.001), presence of tumor rupture (0.6858, p<0.001), intraoperative tumor spill (0.6493, p<0.001), and blood vessel involvement (0.6470, p<0.001). Variables that had lower correlation were determination of the presence of peritoneal implants (0.2753, p<0.001) and interpretation of residual disease status (0.5310, p<0.001).


CONCLUSIONS
The inter-rater reliability of the surgical review is high based on the great consistency in the 3 independent review results. This analysis provides validation and establishes precedent for real-time central surgical review to determine treatment assignment in a risk-based stratagem for multimodal cancer therapy.
Dana Farber Cancer Institute, Boston Children's Hospital, Boston MA. Electronic address: thomas.hamilton@childrens.harvard.edu.
Thomas E
TE
Hamilton
Primary Children's Medical Center, Salt Lake City UT.
Douglas
D
Barnhart
Seattle Children's Hospital, Seattle WA.
Kenneth
K
Gow
Connecticut Children's Hospital, Hartford CT.
Fernando
F
Ferrer
Columbia University Medical Center, New York NY.
Jessica
J
Kandel
Cohen Children's Medical Center of New York, New York NY.
Richard
R
Glick
Cinncinnati Children's Hospital, Cincinnati OH.
Roshni
R
Dasgupta
Children's Oncology Group, University of Florida, Gainesville FL.
Arlene
A
Naranjo
Children's Oncology Group, University of Florida, Gainesville FL.
Ying
Y
He
T. C. Thompson Children's Hospital, Chattanooga TN.
Eric
E
Gratias
Cincinnati Children's Hospital, Cincinnati Medical Center, Cincinnati OH.
James
J
Geller
Dana Farber Cancer Institute, Boston Children's Hospital, Boston MA.
Elizabeth
E
Mullen
C.S. Mott Children's Hospital, University of Michigan, Ann Arbor MI.
Peter
P
Ehrlich
D001706:Biopsy; D003131:Combined Modality Therapy; D003199:Computer Systems; D006801:Humans; D007431:Intraoperative Complications / Q000453:epidemiology; D007680:Kidney Neoplasms / Q000473:pathology / Q000628:therapy*; D008207:Lymphatic Metastasis; D008485:Medical Audit; D009367:Neoplasm Staging; D018365:Neoplasm, Residual; D009389:Neovascularization, Pathologic / Q000453:epidemiology; D009392:Nephrectomy / Q000379:methods / Q000706:statistics & numerical data*; D015588:Observer Variation; D010534:Peritoneal Neoplasms / Q000556:secondary; D015203:Reproducibility of Results; D018570:Risk Assessment; D012422:Rupture, Spontaneous; D009396:Wilms Tumor / Q000473:pathology / Q000556:secondary / Q000628:therapy*
Outcomes; Quality assurance; Surgery; Wilms Tumor