Accuracy of clinicians and models for estimating the probability that a pulmonary nodule is malignant.
BACKGROUND
Management of pulmonary nodules depends critically on the probability of malignancy. Models to estimate probability have been developed and validated, but most clinicians rely on judgment.


OBJECTIVE
The aim of this study was to compare the accuracy of clinical judgment with that of two prediction models.


METHODS
Physician participants reviewed up to five clinical vignettes, selected at random from a larger pool of 35 vignettes, all based on actual patients with lung nodules of known final diagnosis. Vignettes included clinical information and a representative slice from computed tomography. Clinicians estimated the probability of malignancy for each vignette. To examine agreement with models, we calculated intraclass correlation coefficients (ICC) and kappa statistics. To examine accuracy, we compared areas under the receiver operator characteristic curve (AUC).


RESULTS
Thirty-six participants completed 179 vignettes, 47% of which described patients with malignant nodules. Agreement between participants and models was fair for the Mayo Clinic model (ICC, 0.37; 95% confidence interval [CI], 0.23-0.50) and moderate for the Veterans Affairs model (ICC, 0.46; 95% CI, 0.34-0.57). There was no difference in accuracy between participants (AUC, 0.70; 95% CI, 0.62-0.77) and the Mayo Clinic model (AUC, 0.71; 95% CI, 0.62-0.80; P = 0.90) or the Veterans Affairs model (AUC, 0.72; 95% CI, 0.64-0.80; P = 0.54).


CONCLUSIONS
In this vignette-based study, clinical judgment and models appeared to have similar accuracy for lung nodule characterization, but agreement between judgment and the models was modest, suggesting that qualitative and quantitative approaches may provide complementary information.
1 Division of Pulmonary and Critical Care Medicine, Keck School of Medicine of the University of Southern California, Los Angeles, California.
Alex A
AA
Balekian
Gerard A
GA
Silvestri
Suzanne M
SM
Simkovich
Peter J
PJ
Mestaz
Gillian D
GD
Sanders
Jamie
J
Daniel
Jackie
J
Porcel
Michael K
MK
Gould
D002983:Clinical Competence; D003661:Decision Support Techniques; D005260:Female; D006801:Humans; D008175:Lung Neoplasms / Q000175:diagnosis*; D008297:Male; D058007:Physicians, Primary Care / Q000592:standards*; D011336:Probability; D015272:Pulmonary Medicine / Q000592:standards*; D003074:Solitary Pulmonary Nodule / Q000175:diagnosis*