title: Profiling Individual Surgeon Performance Using Information from a High-Quality Clinical Registry: Opportunities and Limitations.
abstract: BACKGROUND
  There is increasing interest in profiling the quality of individual medical providers. Valid assessment of individuals should highlight improvement opportunities, but must be considered in the context of limitations.
  
  
  METHODS
  High quality clinical data from the American College of Surgeons NSQIP, gathered in accordance with strict policies and specifications, was used to construct individual surgeon-level assessments. There were 39,976 cases evaluated, performed by 197 surgeons across 9 hospitals. Both 2-level (cases by surgeon) and 3-level (cases by surgeon by hospital) risk-adjusted, hierarchical regression analyses were performed. Outcomes were 30-day postoperative morbidity, surgical site infection, and mortality. Surgeon performance was compared in both absolute and relative terms. "Signal-to-noise" reliability was calculated for surgeons and models. Projected case requirements for reliability levels were generated.
  
  
  RESULTS
  Surgeon performances could be distinguished to different degrees: morbidity distinguished best, mortality least. Outliers could be identified for morbidity and infection, but not mortality. Reliability was also highest for morbidity and lowest for mortality. Even models with high overall reliability did not assess all providers reliably. Incorporating institutional effects had predictable effects: penalizing providers at "good" institutions, benefiting providers at "poor" institutions.
  
  
  CONCLUSIONS
  Individual surgeon profiles can, at times, be distinguished with moderate or good reliability, but to different degrees in different models. Absolute and relative comparisons are feasible. Incorporating institutional level effects in individual provider modeling presents an interesting policy dilemma, appearing to benefit providers at "poor-performing" institutions, but penalizing those at "high-performing" ones. No portrayal of individual medical provider quality should be accepted without consideration of modeling rationale and, critically, reliability.
authors.affiliation: Department of Surgery, Washington University in St Louis; the Center for Health Policy at Washington University in St Louis; St. Louis Veterans Affairs Medical Center; and BJC Healthcare, St Louis, MO; Olin Business School at Washington University in St Louis, St Louis, MO; Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL. Electronic address: hallb@wustl.edu.
authors.forename: Bruce Lee
authors.initials: BL
authors.lastname: Hall
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL.
authors.forename: Kristopher M
authors.initials: KM
authors.lastname: Huffman
authors.affiliation: Olin Business School at Washington University in St Louis, St Louis, MO.
authors.forename: Barton H
authors.initials: BH
authors.lastname: Hamilton
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, University of Chicago Pritzker School of Medicine, Chicago, IL.
authors.forename: Jennifer L
authors.initials: JL
authors.lastname: Paruch
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL.
authors.forename: Lynn
authors.initials: L
authors.lastname: Zhou
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL.
authors.forename: Karen E
authors.initials: KE
authors.lastname: Richards
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL.
authors.forename: Mark E
authors.initials: ME
authors.lastname: Cohen
authors.affiliation: Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, IL; Department of Surgery, University of California Los Angeles David Geffen School of Medicine and the VA Greater Los Angeles Healthcare System, Los Angeles, CA.
authors.forename: Clifford Y
authors.initials: CY
authors.lastname: Ko
mesh_terms: D019985:Benchmarking / Q000379:methods*; D002983:Clinical Competence / Q000592:standards*; D006801:Humans; D015233:Models, Statistical; D011183:Postoperative Complications / Q000453:epidemiology; D058996:Quality Improvement; D019984:Quality Indicators, Health Care; D012042:Registries; D015203:Reproducibility of Results; D020379:Risk Adjustment; D066231:Surgeons / Q000592:standards*; D014481:United States