title: Rates of false flagging due to statistical artifact in CMS evaluations of transplant programs: results of a stochastic simulation.
abstract: The recent CMS conditions of participation are based on risk-adjusted models produced by the Scientific Registry for Transplant Recipients (SRTR). The accuracy of these models in identifying poor-performing centers is unknown. In this stochastic simulation study, 1-year mortality outcomes were simulated in virtual transplant centers, and used to flag centers according to the methods used by CMS, evaluating nine overlapping 2.5-year periods of simulated data. In a simulation where all centers had the same underlying risk, 10.2% were falsely flagged at least once during the 4.5 years of simulated evaluations. The probability of false-positive flagging was lowest in low-volume centers (2.5%) and highest in high-volume centers (16.2%). In another simulation where 5% of centers were assigned twofold risk ("poor-performing centers"), only 32% of poor-performing centers were correctly flagged. In a final simulation where each center was assigned a unique mortality risk, 94% of flagged centers had greater-than-median risk, but only 32% of flagged centers were among the 5% with highest risk. Even after disregarding known covariate limitations to the risk adjustment models, statistical noise alone leads to spurious flagging of many adequately-performing transplant centers, yet the methods used by CMS fail to flag most centers with true elevated risk.
authors.affiliation: Department of Surgery, Johns Hopkins University School of Medicine, Baltimore, MD, USA.
authors.forename: A B
authors.initials: AB
authors.lastname: Massie
authors.forename: D L
authors.initials: DL
authors.lastname: Segev
mesh_terms: D016477:Artifacts; D014487:Centers for Medicare and Medicaid Services, U.S. / Q000331:legislation & jurisprudence*; D003198:Computer Simulation; D003627:Data Interpretation, Statistical; D006085:Graft Survival; D006801:Humans; D008484:Medicaid / Q000331:legislation & jurisprudence; D006278:Medicare / Q000331:legislation & jurisprudence; D016377:Organ Transplantation / Q000191:economics / Q000401:mortality* / Q000592:standards; D015397:Program Evaluation / Q000706:statistics & numerical data*; D015996:Survival Rate; D009927:Tissue and Organ Procurement; D014481:United States
keywords: Center for Medicare/Medicaid Services; multiple outcomes; policy; transplant outcomes