Assessing the likelihood of an important clinical outcome: new insights from a comparison of clinical and actuarial judgment.
OBJECTIVE
To assess and rank the performance of different methods of predicting the probability of death following a specified surgical procedure.


METHODS
Actuarial estimates of the probability of early mortality for 40 patients were derived from 2 sources: a large published surgical series and a smaller series from the center where surgery was performed. Surgeons and cardiologists also provided probability estimates for these patients.


RESULTS
Estimates derived from the published literature were too optimistic and did not differentiate between patients more, or less, likely to die (i.e., failed to discriminate). Doctors' judgments were unbiased but failed to discriminate. Local actuarial estimates (influenced by only 1 or 2 variables) were unbiased, did discriminate, but exhibited more random variation.


CONCLUSIONS
The preferred source of estimates depends upon which aspect of accuracy is of greatest importance. Differences in patient selection and error in the identification of risk factors mean that published results will not always appropriately predict surgical risk at other institutions. Risk stratification may be more robust when based on a small set of cross-validated predictors rather than a larger set of predictors that includes some whose reliability has not been confirmed.
Rakow
Tim
T
Department of Psychology, University of Essex, Colchester, UK. timrakow@essex.ac.uk
Vincent
Charles
C
Bull
Kate
K
Harvey
Nigel
N
D000206:Actuarial Analysis; D000367:Age Factors; D002675:Child, Preschool; D003661:Decision Support Techniques; D018729:Fontan Procedure / Q000401:mortality*; D006330:Heart Defects, Congenital / Q000601:surgery*; D006352:Heart Ventricles / Q000002:abnormalities / Q000601:surgery*; D017052:Hospital Mortality; D006801:Humans; D007223:Infant; D007231:Infant, Newborn; D007600:Judgment; D016015:Logistic Models; D015999:Multivariate Analysis; D018579:Patient Selection; D011379:Prognosis; D016016:Proportional Hazards Models; D015203:Reproducibility of Results; D018570:Risk Assessment / Q000379:methods / Q000706:statistics & numerical data*; D012307:Risk Factors; D016019:Survival Analysis; D016896:Treatment Outcome