title: Use and misuse of waterfall plots.
abstract: BACKGROUND
  "Waterfall plots" are used to describe changes in tumor size observed in clinical studies. Here we assess criteria for generation of waterfall plots and the impact of measurement error in generating them.
  
  
  METHODS
  We reviewed published waterfall plots to investigate variability in criteria used to define them. We then compared waterfall plots generated by different observers for 24 patients enrolled in a completed phase I study of solid tumors with available computed tomography (CT) scans. Tumor measurements were made independently from CT scans according to Response Evaluation Criteria in Solid Tumors 1.1 by four board-certified radiologists and four medical oncologists. Interobserver variability was quantified and compared with reference measurements reported for the phase 1 study. All statistical tests were two-sided.
  
  
  RESULTS
  There was substantial variability in criteria used to generate published waterfall plots. In the internal study, the results were statistically significantly different between all eight readers (P = .01, variance = 197.1, SD = 14.0) and between the oncologists (P = .01, variance = 319.0, SD = 17.9), but not between the radiologists (P = .68, variance = 70.8, SD = 8.4). Different observers classified one to five patients as having a partial response and 12-19 patients as having stable disease. Similar variability in categorization of response was observed when these error rates were applied to published waterfall plots.
  
  
  CONCLUSIONS
  Waterfall plots are subject to substantial variability in criteria used to define them and are influenced by measurement errors; they should be generated by trained radiologists. Caution should be exercised when interpreting results of waterfall plots in the context of clinical trials.
authors.affiliation: Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS).
authors.forename: Tiffany
authors.initials: T
authors.lastname: Shao
authors.affiliation: Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS).
authors.forename: Lisa
authors.initials: L
authors.lastname: Wang
authors.affiliation: Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS).
authors.forename: Arnoud J
authors.initials: AJ
authors.lastname: Templeton
authors.affiliation: Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS).
authors.forename: Raymond
authors.initials: R
authors.lastname: Jang
authors.affiliation: Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS).
authors.forename: Francisco W
authors.initials: FW
authors.lastname: Vera-Badillo
authors.affiliation: Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS).
authors.forename: Mairead G
authors.initials: MG
authors.lastname: McNamara
authors.affiliation: Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS).
authors.forename: Myles
authors.initials: M
authors.lastname: Margolis
authors.affiliation: Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS).
authors.forename: Tae Kyoung
authors.initials: TK
authors.lastname: Kim
authors.affiliation: Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS).
authors.forename: Mehrdad
authors.initials: M
authors.lastname: Sinaei
authors.affiliation: Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS).
authors.forename: Hassan
authors.initials: H
authors.lastname: Shoushtari
authors.affiliation: Divisions of Medical Oncology and Hematology (TS, AJT, RWJ, FVB, MGM, IFT) and Biostatistics (LW), Princess Margaret Cancer Center, Department of Medicine, University of Toronto, Toronto, Canada; Joint Department of Medical Imaging, University Health Network, Toronto, Canada (MM, TKK, MS, HS). ian.tannock@uhn.ca.
authors.forename: Ian F
authors.initials: IF
authors.lastname: Tannock
mesh_terms: D053799:2-Pyridinylmethylsulfinylbenzimidazoles / Q000008:administration & dosage; D000971:Antineoplastic Combined Chemotherapy Protocols / Q000627:therapeutic use*; D017321:Clinical Trials, Phase I as Topic / Q000379:methods; D003627:Data Interpretation, Statistical; D004317:Doxorubicin / Q000008:administration & dosage; D006801:Humans; D009369:Neoplasms / Q000000981:diagnostic imaging / Q000188:drug therapy / Q000473:pathology*; D015588:Observer Variation; D009864:Ontario; D000077402:Pantoprazole; D014057:Tomography, X-Ray Computed; D016896:Treatment Outcome
chemical_list: D053799:2-Pyridinylmethylsulfinylbenzimidazoles; D004317:Doxorubicin; D000077402:Pantoprazole