Developing and testing a model to predict outcomes of organizational change.
OBJECTIVE
To test the effectiveness of a Bayesian model employing subjective probability estimates for predicting success and failure of health care improvement projects.


METHODS
Experts' subjective assessment data for model development and independent retrospective data on 221 healthcare improvement projects in the United States, Canada, and The Netherlands collected between 1996 and 2000 for validation.


METHODS
A panel of theoretical and practical experts and literature in organizational change were used to identify factors predicting the outcome of improvement efforts. A Bayesian model was developed to estimate probability of successful change using subjective estimates of likelihood ratios and prior odds elicited from the panel of experts. A subsequent retrospective empirical analysis of change efforts in 198 health care organizations was performed to validate the model. Logistic regression and ROC analysis were used to evaluate the model's performance using three alternative definitions of success.


METHODS
For the model development, experts' subjective assessments were elicited using an integrative group process. For the validation study, a staff person intimately involved in each improvement project responded to a written survey asking questions about model factors and project outcomes.


RESULTS
Logistic regression chi-square statistics and areas under the ROC curve demonstrated a high level of model performance in predicting success. Chi-square statistics were significant at the 0.001 level and areas under the ROC curve were greater than 0.84.


CONCLUSIONS
A subjective Bayesian model was effective in predicting the outcome of actual improvement projects. Additional prospective evaluations as well as testing the impact of this model as an intervention are warranted.
Gustafson
David H
DH
University of Wisconsin-Madison, USA.
Sainfort
François
F
Eichler
Mary
M
Adams
Laura
L
Bisognano
Maureen
M
Steudel
Harold
H
D001499:Bayes Theorem; D002170:Canada; D016009:Chi-Square Distribution; D006302:Health Services Research / Q000379:methods*; D006801:Humans; D016015:Logistic Models; D009426:Netherlands; D009936:Organizational Innovation; D011237:Predictive Value of Tests; D015397:Program Evaluation / Q000379:methods*; D012372:ROC Curve; D012189:Retrospective Studies; D017749:Total Quality Management / Q000458:organization & administration*; D014481:United States