title: How do you tell whether a change in surgical technique leads to a change in outcome?
abstract: OBJECTIVE
  Surgeons routinely evaluate and modify their surgical technique to improve patient outcome. It is also common for surgeons to analyze results before and after a change in technique to determine whether the change led to better results. Simple comparison of results before and after surgical modification may be confounded by the surgical learning curve. We developed a statistical method applicable to analyzing before/after surgical studies.
  
  
  METHODS
  We used simulation studies to compare different statistical analyses of before/after studies. We evaluated a simple 2-group comparison of results before and after the modification by the chi-square test and a novel bootstrap method that adjusts for the surgical learning curve.
  
  
  RESULTS
  In the presence of the learning curve a simple 2-group comparison almost always showed an ineffective surgical modification to be of benefit. When the surgical modification was harmful, leading to a 10% decrease in the success rate, 2-group comparison nonetheless showed a statistically significant improvement in outcome about 80% of the time. The bootstrap method had only moderate power but did not show that ineffective surgical modifications were beneficial more than would be expected by chance.
  
  
  CONCLUSIONS
  Simplistic approaches to the analysis of before/after surgical studies may lead to grossly erroneous results under the surgical learning curve. A straightforward alternative statistical method allows investigators to separate the effects of the learning curve from those of the surgical modification.
authors.affiliation: Department of Epidemiology and Biostatistics and Urology Service, Memorial Sloan-Kettering Cancer Center, New York, New York 10065, USA. vickersa@mskcc.org
authors.forename: Andrew J
authors.initials: AJ
authors.lastname: Vickers
authors.forename: Angel M
authors.initials: AM
authors.lastname: Cronin
authors.forename: Timothy A
authors.initials: TA
authors.lastname: Masterson
authors.forename: James A
authors.initials: JA
authors.lastname: Eastham
mesh_terms: D003627:Data Interpretation, Statistical; D006801:Humans; D008297:Male; D011468:Prostatectomy / Q000193:education / Q000379:methods* / Q000592:standards; D012107:Research Design; D016896:Treatment Outcome