title: Certain performance values arising from mammographic test set readings correlate well with clinical audit.
abstract: BACKGROUND
  Test sets have been increasingly utilised to augment clinical audit in breast screening programmes; however, their relationship has never been satisfactorily understood. This study examined the relationship between mammographic test set performance and clinical audit data.
  
  
  METHODS
  Clinical audit data over a 2-year period was generated for each of 20 radiologists. Sixty mammographic examinations, consisting of 40 normal and 20 cancer cases, formed the test set. Readers located any identifiable cancer, and levels of confidence were scored from 2 to 5, where a score of 3 and above is considered a recall rating. Jackknifing free response operating characteristic (JAFROC) figure-of-merit (FOM), location sensitivity and specificity were calculated for individual readers and then compared with clinical audit values using Spearman's rho.
  
  
  RESULTS
  JAFROC FOM showed significant correlations to: recall rate at a first round of screening (r = 0.51; P = 0.02); rate of small invasive cancers per 10 000 reads (r = 0.5; P = 0.02); percentage of all cancers read that were not recalled (r = -0.51; P = 0.02); and sensitivity (r = 0.51; P = 0.02). Location sensitivity demonstrated significant correlations with: rate of small invasive cancers per 10 000 reads (r = 0.46; P = 0.04); rate of DCIS (ductal carcinoma in situ) per 10 000 reads (r = 0.44; P = 0.05); detection rate of all invasive cancers and DCIS per 10 000 reads (r = 0.54; P = 0.01); percentage of all cancers read that were not recalled (r = -0.57; P = 0.009); and sensitivity (r = 0.57; P = 0.009). No other significant relationships were noted.
  
  
  CONCLUSIONS
  Performance indicators from test set demonstrate significant correlations with specific aspects of clinical performance, although caution needs to be exercised when generalising test set specificity to the clinical situation.
authors.affiliation: Department of Diagnostic Radiology, Singapore General Hospital, Singapore.
authors.forename: BaoLin Pauline
authors.initials: BP
authors.lastname: Soh
authors.affiliation: Cancer Institute NSW, Sydney, New South Wales, Australia.
authors.forename: Warwick Bruce
authors.initials: WB
authors.lastname: Lee
authors.affiliation: Medical Image Optimisation and Perception Group (MIOPeG), Discipline of Medical Radiation Sciences (C42), University of Sydney, Sydney, New South Wales, Australia.
authors.forename: Claudia
authors.initials: C
authors.lastname: Mello-Thoms
authors.affiliation: Medical Image Optimisation and Perception Group (MIOPeG), Discipline of Medical Radiation Sciences (C42), University of Sydney, Sydney, New South Wales, Australia.
authors.forename: Kriscia
authors.initials: K
authors.lastname: Tapia
authors.affiliation: Ziltron, Dublin, Ireland.
authors.forename: John
authors.initials: J
authors.lastname: Ryan
authors.affiliation: Cancer Institute NSW, Sydney, New South Wales, Australia.
authors.forename: Wai Tak
authors.initials: WT
authors.lastname: Hung
authors.affiliation: Cancer Institute NSW, Sydney, New South Wales, Australia.
authors.forename: Graham
authors.initials: G
authors.lastname: Thompson
authors.affiliation: Medical Image Optimisation and Perception Group (MIOPeG), Discipline of Medical Radiation Sciences (C42), University of Sydney, Sydney, New South Wales, Australia.
authors.forename: Rob
authors.initials: R
authors.lastname: Heard
authors.affiliation: Medical Image Optimisation and Perception Group (MIOPeG), Discipline of Medical Radiation Sciences (C42), University of Sydney, Sydney, New South Wales, Australia.
authors.forename: Patrick
authors.initials: P
authors.lastname: Brennan
mesh_terms: D001315:Australia; D001943:Breast Neoplasms / Q000000981:diagnostic imaging*; D002983:Clinical Competence / Q000592:standards* / Q000706:statistics & numerical data*; D055088:Early Detection of Cancer / Q000592:standards / Q000706:statistics & numerical data; D005260:Female; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000592:standards; D008327:Mammography / Q000592:standards* / Q000706:statistics & numerical data*; D008485:Medical Audit; D015588:Observer Variation; D011785:Quality Assurance, Health Care / Q000592:standards; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D013223:Statistics as Topic
keywords: breast; clinical audit; mammography; observer performance; test set
references.citation: Jemal A, Siegel R, Ward E, Hao Y, Xu J, Thun MJ. Cancer statistics, 2009. CA Cancer J Clin 2009; 59: 225-249.
references.citation: Australian Institute of Health and Welfare (AIHW), Cancer Australia, Australasian Association of Cancer Registries. Cancer survival and prevalence in Australia: cancers diagnosed from 1982 to 2004. Cancer Series no. 42. Cat. no. CAN 38. Canberra: AIHW; 2008.
references.citation: Australian Institute of Health and Welfare. BreastScreen Australia monitoring report 2010-2011. 2013. [Cited 4 Jun 2014.] Available from URL: http://www.aihw.gov.au/WorkArea/DownloadAsset.aspx?id=60129544880.
references.citation: Barlow WE, Chi C, Carney PA et al. Accuracy of screening mammography interpretation by characteristics of radiologists. J Natl Cancer Inst 2004; 96: 1840-1850.
references.citation: Sickles EA, Wolverton DE, Dee KE. Performance parameters for screening and diagnostic mammography: specialist and general radiologists. Radiology 2002; 224: 861-869.
references.citation: Beam CA, Layde PM, Sullivan DC. Variability in the interpretation of screening mammograms by US radiologists. Findings from a national sample. Arch Intern Med 1996; 156: 209-213.
references.citation: Elmore JG, Wells CK, Lee CH, Howard DH, Feinstein AR. Variability in radiologists' interpretations of mammograms. N Engl J Med 1994; 331: 1493-1499.
references.citation: Kessler LG, Andersen MR, Etzioni R. Much ado about mammography variability. J Natl Cancer Inst 2002; 94: 1346-1347.
references.citation: Elmore JG, Miglioretti DL, Reisch LM et al. Screening mammograms by community radiologists: variability in false-positive rates. J Natl Cancer Inst 2002; 94: 1373-1380.
references.citation: Soh BP, Lee W, Kench PL et al. Assessing reader performance in radiology, an imperfect science: lessons from breast screening. Clin Radiol 2012; 67: 623-628.
references.citation: Brennan PC, Tapia K, Lee W. BreastScreen Reader Assessment Strategy (BREAST). University of Sydney, Sydney, 2013. [Cited 19 Oct 2013.]. Available from URL: http://sydney.edu.au/health-sciences/breastaustralia/.
references.citation: Gale A. PERFORMS: a self assessment scheme for radiologists in breast screening. Semin Breast Dis 2003; 6: 148-152.
references.citation: Scott HJ, Gale AG. Breast screening: PERFORMS identifies key mammographic training needs. Br J Radiol 2006; 79: S127-133.
references.citation: Scott HJ, Evan A, Gale AG, Murphy A, Reed J. The relationship between real life breast screening and an annual self assessment scheme. Proc SPIE 2009; 7263: 72631E-1-7.
references.citation: Gale AG. Maintaining quality in the UK breast screening program. Proc SPIE 2010; 7627: 762702-1-2.
references.citation: McEntee M, Brennan P, Evanoff M, Philips P, O Connor WT, Manning D. Optimum ambient lighting conditions for the viewing of softcopy radiological images. Proc SPIE 2006; 6146: 260-268.
references.citation: Brennan PC, McEntee M, Evanoff M, Philips P, O'Connor WT, Manning DJ. Ambient lighting: effect of illumination on soft-copy viewing of radiographs of the wrist. AJR Am J Roentgenol 2007; 188: 177-180.
references.citation: Soh BP, Lee W, Diffey JL et al. Breast Screen New South Wales generally demonstrates good radiologic viewing conditions. J Digit Imaging 2013; 26: 759-767.
references.citation: Levin J, Fox J. Elementary Statistics in Social Research. Harper Collins Publishers, New York, 1994.
references.citation: Soh BP, Lee W, McEntee MF et al. Screening mammography: test set data can reasonably describe actual clinical reporting. Radiology 2013; 268: 46-53.
references.citation: Rutter CM, Taplin S. Assessing mammographers' accuracy: a comparison of clinical and test performance. J Clin Epidemiol 2000; 53: 443-450.
references.citation: Gur D, Bandos AL, Cohen CS et al. The ‘laboratory’ effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations. Radiology 2008; 249: 47-53.
references.citation: BreastScreen Australia National Accreditation Standards. BreastScreen Australia quality improvement program. 2008. [Cited 15 Dec 2014.] Available from URL: http://www.cancerscreening.gov.au/internet/screening/publishing.nsf/Content/br-accreditation/$File/standards.pdf.
references.citation: Egglin TK, Feinstein AR. Context bias. A problem in diagnostic radiology. J Am Med Assoc 1996; 276: 1752-1755.