How do you tell whether a change in surgical technique leads to a change in outcome?
OBJECTIVE
Surgeons routinely evaluate and modify their surgical technique to improve patient outcome. It is also common for surgeons to analyze results before and after a change in technique to determine whether the change led to better results. Simple comparison of results before and after surgical modification may be confounded by the surgical learning curve. We developed a statistical method applicable to analyzing before/after surgical studies.


METHODS
We used simulation studies to compare different statistical analyses of before/after studies. We evaluated a simple 2-group comparison of results before and after the modification by the chi-square test and a novel bootstrap method that adjusts for the surgical learning curve.


RESULTS
In the presence of the learning curve a simple 2-group comparison almost always showed an ineffective surgical modification to be of benefit. When the surgical modification was harmful, leading to a 10% decrease in the success rate, 2-group comparison nonetheless showed a statistically significant improvement in outcome about 80% of the time. The bootstrap method had only moderate power but did not show that ineffective surgical modifications were beneficial more than would be expected by chance.


CONCLUSIONS
Simplistic approaches to the analysis of before/after surgical studies may lead to grossly erroneous results under the surgical learning curve. A straightforward alternative statistical method allows investigators to separate the effects of the learning curve from those of the surgical modification.
Vickers
Andrew J
AJ
Department of Epidemiology and Biostatistics and Urology Service, Memorial Sloan-Kettering Cancer Center, New York, New York 10065, USA. vickersa@mskcc.org
Cronin
Angel M
AM
Masterson
Timothy A
TA
Eastham
James A
JA
D003627:Data Interpretation, Statistical; D006801:Humans; D008297:Male; D011468:Prostatectomy / Q000193:education / Q000379:methods* / Q000592:standards; D012107:Research Design; D016896:Treatment Outcome