The prevalence effect in a laboratory environment: Changing the confidence ratings.
OBJECTIVE
We sought to assess whether or not prevalence levels affected the confidence ratings of readers during the interpretation of cases in a laboratory receiver operating characteristic-type observer performance study.


METHODS
We reanalyzed a previously conducted observer performance study that included 14 readers and 5 different levels of prevalence. The previous study yielded the observation that in the laboratory we could not detect a "prevalence effect" in terms of differences in areas under the receiver operating characteristic curves. The detection ratings (for presence or absence) of lung nodules, interstitial disease, and pneumothorax for the five prevalence levels were compared, and a test for trend in averaged ratings as a function of abnormality prevalence was performed within a mixed-model setting that accounts for different sources of variability and correlations induced by the study design.


RESULTS
The ratings of the cases in terms of confidence that the specific abnormality in question is present tend, on average, to be larger when actual disease prevalence is lower. The rate of the increase of the average confidence ratings with the decreasing prevalence of a specific abnormality is very similar for actually positive and actually negative cases for every considered abnormality. The observed trend in the changes of the average confidence ratings as a function of prevalence levels was statistically significant (p < 0.01).


CONCLUSIONS
Expectations of disease prevalence in the case mix during a laboratory observer performance study may systematically affect the behavior of observers in terms of their actual confidence ratings.
Gur
David
D
Department of Radiology, Imaging Research, Suite 4200 Magee-Womens Hospital, 300 Halket Street, School of Medicine, Pittsburgh, PA 15213-3180, USA. gurd@upmc.edu
Bandos
Andriy I
AI
Fuhrman
Carl R
CR
Klym
Amy H
AH
King
Jill L
JL
Rockette
Howard E
HE
D006801:Humans; D007753:Laboratories; D009770:Obsessive Behavior; D015995:Prevalence; D012372:ROC Curve; D011859:Radiography