Observer performance studies: detection of single versus multiple abnormalities of the chest.
OBJECTIVE
We used receiver operating characteristic (ROC) analysis to compare two methods of evaluating observer performance in detecting an abnormality on chest radiographs. In the first method, the abnormality in question, rib fracture, was one of five investigated, and it was the only one of interest in the second.


METHODS
Eight experienced observers viewed 117 posteroanterior chest radiographs in two interpretation modes. Fifty-four of these images depicted rib fractures that had been rated as subtle for detection. The likelihood of the presence of a rib fracture was rated as one of five abnormalities in question in one mode and the sole abnormality of interest in the other mode.


RESULTS
Six of the observers performed better during the single-abnormality mode, one performed equally well in both modes, and one performed better during the multiple-abnormality mode. The average area under the ROC curves (A(z)) was 0.73 +/- 0.07 for the multiple-abnormality mode and 0.80 +/- 0.04 for the single-abnormality mode. The results were significantly different (p < 0.05).


CONCLUSIONS
Study methodology can significantly affect the results in ROC studies, particularly for abnormalities that may not be perceived as primary or important. The order in which abnormalities appear on a checklist report form may be important.
Fuhrman
Carl R
CR
Department of Radiology, University of Pittsburgh, 200 Lothrop St., Pittsburgh, PA 15213-2582, USA.
Britton
Cynthia A
CA
Bender
Thomas
T
Sumkin
Jules H
JH
Brown
Manuel L
ML
Holbert
J Michael
JM
Chang
Thomas S
TS
Rockette
Howard E
HE
Gur
David
D
D019540:Area Under Curve; D006801:Humans; D008171:Lung Diseases / Q000150:complications / Q000000981:diagnostic imaging*; D015588:Observer Variation; D012372:ROC Curve; D013902:Radiography, Thoracic; D012253:Rib Fractures / Q000150:complications / Q000000981:diagnostic imaging*