title: Validity and Reliability of Value Assessment Frameworks for New Cancer Drugs.
abstract: Several organizations have developed frameworks to systematically assess the value of new drugs. These organizations include the American Society of Clinical Oncology (ASCO), the European Society for Medical Oncology (ESMO), the Institute for Clinical and Economic Review (ICER), and the National Comprehensive Cancer Network (NCCN).
  
  
  
  To understand the extent to which these four tools can facilitate value-based treatment decisions in oncology.
  
  
  
  In this pilot study, eight panelists conducted value assessments of five advanced lung cancer drugs using the ASCO, ESMO, and ICER frameworks. The panelists received instructions and published clinical data required to complete the assessments. Published NCCN framework scores were abstracted. The Kendall's W coefficient was used to measure convergent validity among the four frameworks. Intraclass correlation coefficients were used to measure inter-rater reliability among the ASCO, ESMO, and ICER frameworks. Sensitivity analyses were conducted.
  
  
  
  Drugs were ranked similarly by the four frameworks, with Kendall's W of 0.703 (P = 0.006) across all the four frameworks. Pairwise, Kendall's W was the highest for ESMO-ICER (W = 0.974; P = 0.007) and ASCO-NCCN (W = 0.944; P = 0.022) and the lowest for ICER-NCCN (W = 0.647; P = 0.315) and ESMO-NCCN (W = 0.611; P = 0.360). Intraclass correlation coefficients (confidence interval [CI]) for the ASCO, ESMO, and ICER frameworks were 0.786 (95% CI 0.517-0.970), 0.804 (95% CI 0.545-0.973), and 0.281 (95% CI 0.055-0.799), respectively. When scores were rescaled to 0 to 100, the ICER framework provided the narrowest band of scores.
  
  
  
  The ASCO, ESMO, ICER, and NCCN frameworks demonstrated convergent validity, despite differences in conceptual approaches used. The ASCO inter-rater reliability was high, although potentially at the cost of user burden. The ICER inter-rater reliability was poor, possibly because of its failure to distinguish differential value among the sample of drugs tested. Refinements of all frameworks should continue on the basis of further testing and stakeholder feedback.
authors.affiliation: Partnership for Health Analytic Research, LLC, Beverly Hills, CA, USA. Electronic address: tbentley@pharllc.com.
authors.forename: Tanya G K
authors.initials: TGK
authors.lastname: Bentley
authors.affiliation: Tufts Medical Center, Boston, MA, USA.
authors.forename: Joshua T
authors.initials: JT
authors.lastname: Cohen
authors.affiliation: Memorial Sloan Kettering Cancer Center, New York, NY, USA.
authors.forename: Elena B
authors.initials: EB
authors.lastname: Elkin
authors.affiliation: Hematology Oncology of San Fernando Valley, Encino, CA, USA.
authors.forename: Julie
authors.initials: J
authors.lastname: Huynh
authors.affiliation: Health Sciences Program, California State University, East Bay, Hayward, CA, USA.
authors.forename: Arnab
authors.initials: A
authors.lastname: Mukherjea
authors.affiliation: Department of Medicine, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA.
authors.forename: Thanh H
authors.initials: TH
authors.lastname: Neville
authors.affiliation: City of Hope National Medical Center, Duarte, CA, USA.
authors.forename: Matthew
authors.initials: M
authors.lastname: Mei
authors.affiliation: Eisai Inc., Woodcliff Lake, NJ, USA.
authors.forename: Ronda
authors.initials: R
authors.lastname: Copher
authors.affiliation: Eisai Inc., Woodcliff Lake, NJ, USA.
authors.forename: Russell
authors.initials: R
authors.lastname: Knoth
authors.affiliation: Department of Medicine, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA.
authors.forename: Ioana
authors.initials: I
authors.lastname: Popescu
authors.affiliation: Partnership for Health Analytic Research, LLC, Beverly Hills, CA, USA.
authors.forename: Jackie
authors.initials: J
authors.lastname: Lee
authors.affiliation: Partnership for Health Analytic Research, LLC, Beverly Hills, CA, USA.
authors.forename: Jenelle M
authors.initials: JM
authors.lastname: Zambrano
authors.affiliation: Partnership for Health Analytic Research, LLC, Beverly Hills, CA, USA.
authors.forename: Michael S
authors.initials: MS
authors.lastname: Broder
mesh_terms: D000970:Antineoplastic Agents / Q000592:standards*; D003661:Decision Support Techniques; D008495:Medical Oncology; D010865:Pilot Projects; D015203:Reproducibility of Results; D062505:Value-Based Purchasing
chemical_list: D000970:Antineoplastic Agents
keywords: convergent validity; inter-rater reliability; oncology; value frameworks