title: Reliability of the American College of Surgeons Commission on Cancer's Quality of Care Measures for Hospital and Surgeon Profiling.
abstract: BACKGROUND
  Efforts to improve healthcare quality involve profiling hospitals and providers. Whether cancer-specific measures can be used reliably for profiling purposes has not been reported.
  
  
  METHODS
  Hospitals and surgeons were profiled with 3 measures assessing the adequacy of lymphadenectomy for colon (ie at least 12 regional lymph nodes [12RLN] are removed and pathologically examined for resected colon cancer), gastric (ie at least 15 regional lymph nodes [G15RLN] are removed and pathologically examined for resected gastric cancer), and non-small cell lung (ie at least 10 regional lymph nodes [10RLN] are removed and pathologically examined for American Joint Committee on Cancer stage IA, IB, IIA, and IIB resected non-small cell lung cancer) cancers using hierarchical models. National Cancer Data Base cases spanning 2010 to 2013 were included if they met measure eligibility. Reliability estimates for hospital and surgeon performance across cumulative years of data (2013, 2012 to 2013, 2011 to 2013, and 2010 to 2013) were calculated with and without risk adjustment. Surgeon caseload minimums were projected to achieve reliabilities of 0.40 and 0.70.
  
  
  RESULTS
  Reliability estimates tended to increase with longer periods of data collection but at different rates, depending on measure, level of aggregation, and performance outlier status. Profiling hospitals using 12RLN with 2 years of data yielded a median reliability of 0.72 (interquartile range [IQR] 0.55 to 0.83); however, 4 years of data yielded a median reliability of only 0.31 (IQR 0.14 to 0.54) for surgeons. The G15RLN performance was poor overall; 10RLN had high reliability at both hospital (0.74; IQR 0.50 to 0.86) and surgeon (0.61; IQR 0.34 to 0.80) levels using 1 year of data, but the literature questions this measure's validity. Few surgeons could achieve appropriate levels of reliability regardless of increased data collection duration.
  
  
  CONCLUSIONS
  Profiling hospitals based on measures such as these can achieve acceptable reliability in reasonable timeframes, but does not always. Either lower levels of reliability should be accepted to profile surgeons with these measures or longer timeframes should be used.
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, University of Chicago Hospitals, Chicago, IL. Electronic address: jliu@facs.org.
authors.forename: Jason B
authors.initials: JB
authors.lastname: Liu
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL.
authors.forename: Kristopher M
authors.initials: KM
authors.lastname: Huffman
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL.
authors.forename: Bryan E
authors.initials: BE
authors.lastname: Palis
authors.affiliation: Abramson Cancer Center, Department of Medicine, University of Pennsylvania, Philadelphia, PA.
authors.forename: Lawrence N
authors.initials: LN
authors.lastname: Shulman
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL.
authors.forename: David P
authors.initials: DP
authors.lastname: Winchester
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, University of California Los Angeles David Geffen School of Medicine, Los Angeles, CA; VA Greater Los Angeles Healthcare System, Los Angeles, CA.
authors.forename: Clifford Y
authors.initials: CY
authors.lastname: Ko
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, Washington University in St Louis, St Louis, MO; Center for Health Policy and the Olin Business School, Washington University in St Louis, St Louis, MO; Saint Louis Veterans Affairs Medical Center, St Louis, MO; BJC Healthcare, St Louis, MO.
authors.forename: Bruce L
authors.initials: BL
authors.lastname: Hall
mesh_terms: D000328:Adult; D000368:Aged; D000369:Aged, 80 and over; D016208:Databases, Factual; D005260:Female; D006761:Hospitals / Q000592:standards* / Q000706:statistics & numerical data; D006801:Humans; D016015:Logistic Models; D008197:Lymph Node Excision / Q000592:standards* / Q000706:statistics & numerical data; D008297:Male; D008875:Middle Aged; D009369:Neoplasms / Q000601:surgery*; D011785:Quality Assurance, Health Care / Q000379:methods* / Q000706:statistics & numerical data; D019984:Quality Indicators, Health Care / Q000706:statistics & numerical data*; D015203:Reproducibility of Results; D012189:Retrospective Studies; D012955:Societies, Medical; D066231:Surgeons / Q000592:standards* / Q000706:statistics & numerical data; D014481:United States