Identifying minimally acceptable interpretive performance criteria for screening mammography.
OBJECTIVE
To develop criteria to identify thresholds for minimally acceptable physician performance in interpreting screening mammography studies and to profile the impact that implementing these criteria may have on the practice of radiology in the United States.


METHODS
In an institutional review board-approved, HIPAA-compliant study, an Angoff approach was used in two phases to set criteria for identifying minimally acceptable interpretive performance at screening mammography as measured by sensitivity, specificity, recall rate, positive predictive value (PPV) of recall (PPV(1)) and of biopsy recommendation (PPV(2)), and cancer detection rate. Performance measures were considered separately. In phase I, a group of 10 expert radiologists considered a hypothetical pool of 100 interpreting physicians and conveyed their cut points of minimally acceptable performance. The experts were informed that a physician's performance falling outside the cut points would result in a recommendation to consider additional training. During each round of scoring, all expert radiologists' cut points were summarized into a mean, median, mode, and range; these were presented back to the group. In phase II, normative data on performance were shown to illustrate the potential impact cut points would have on radiology practice. Rescoring was done until consensus among experts was achieved. Simulation methods were used to estimate the potential impact of performance that improved to acceptable levels if effective additional training was provided.


RESULTS
Final cut points to identify low performance were as follows: sensitivity less than 75%, specificity less than 88% or greater than 95%, recall rate less than 5% or greater than 12%, PPV(1) less than 3% or greater than 8%, PPV(2) less than 20% or greater than 40%, and cancer detection rate less than 2.5 per 1000 interpretations. The selected cut points for performance measures would likely result in 18%-28% of interpreting physicians being considered for additional training on the basis of sensitivity and cancer detection rate, while the cut points for specificity, recall, and PPV(1) and PPV(2) would likely affect 34%-49% of practicing interpreters. If underperforming physicians moved into the acceptable range, detection of an additional 14 cancers per 100000 women screened and a reduction in the number of false-positive examinations by 880 per 100000 women screened would be expected.


CONCLUSIONS
This study identified minimally acceptable performance levels for interpreters of screening mammography studies. Interpreting physicians whose performance falls outside the identified cut points should be reviewed in the context of their specific practice settings and be considered for additional training.
Carney
Patricia A
PA
Department of Family Medicine and Department of Public Health and Preventive Medicine, Oregon Health & Science University, 3181 SW Sam Jackson Park Rd, Portland, OR 97239-3098, USA. carneyp@ohsu.edu
Sickles
Edward A
EA
Monsees
Barbara S
BS
Bassett
Lawrence W
LW
Brenner
R James
RJ
Feig
Stephen A
SA
Smith
Robert A
RA
Rosenberg
Robert D
RD
Bogart
T Andrew
TA
Browning
Sally
S
Barry
Jane W
JW
Kelly
Mary M
MM
Tran
Khai A
KA
Miglioretti
Diana L
DL
D001706:Biopsy; D001943:Breast Neoplasms / Q000000981:diagnostic imaging*; D002983:Clinical Competence / Q000592:standards*; D005260:Female; D006801:Humans; D008327:Mammography / Q000592:standards*; D008403:Mass Screening / Q000592:standards*; D011237:Predictive Value of Tests; D011871:Radiology / Q000592:standards*; D012680:Sensitivity and Specificity; D014481:United States