title: Comparison of visual grading and free-response ROC analyses for assessment of image-processing algorithms in digital mammography.
abstract: OBJECTIVE
  To compare two methods for assessment of image-processing algorithms in digital mammography: free-response receiver operating characteristic (FROC) for the specific task of microcalcification detection and visual grading analysis (VGA).
  
  
  METHODS
  The FROC study was conducted prior to the VGA study reported here. 200 raw data files of low breast density (Breast Imaging-Reporting and Data System I-II) mammograms (Novation DR, Siemens, Germany)-100 of which abnormal-were processed by four image-processing algorithms: Raffaello (IMS, Bologna, Italy), Sigmoid (Sectra, Linköping, Sweden), and OpView v. 2 and v. 1 (Siemens, Erlangen, Germany). Four radiologists assessed the mammograms for the detection of microcalcifications. 8 months after the FROC study, a subset (200) of the 800 images was reinterpreted by the same radiologists, using the VGA methodology in a side-by-side approach. The VGA grading was based on noise, saturation, contrast, sharpness and confidence with the image in terms of normal structures. Ordinal logistic regression was applied; OpView v. 1 was the reference processing algorithm.
  
  
  RESULTS
  In the FROC study all algorithms performed better than OpView v. 1. From the current VGA study and for confidence with the image, Sigmoid and Raffaello were significantly worse (p<0.001) than OpView v. 1; OpView v. 2 was significantly better (p=0.01). For the image quality criteria, results were mixed; Raffaello and Sigmoid for example were better than OpView v. 1 for sharpness and contrast (although not always significantly).
  
  
  CONCLUSIONS
  VGA and FROC discordant results should be attributed to the different clinical task addressed.
  
  
  CONCLUSIONS
  The method to use for image-processing assessment depends on the clinical task tested.
authors: Leuven University Center of Medical Physics in Radiology, University Hospitals Leuven, Belgium. federica.zanca@uzleuven.be
authors: F
authors: F
authors: Zanca
authors: C
authors: C
authors: Van Ongeval
authors: F
authors: F
authors: Claus
authors: J
authors: J
authors: Jacobs
authors: R
authors: R
authors: Oyen
authors: H
authors: H
authors: Bosmans
mesh_terms: D000465:Algorithms; D001941:Breast Diseases / Q000000981:diagnostic imaging*; D001943:Breast Neoplasms / Q000000981:diagnostic imaging; D002114:Calcinosis / Q000000981:diagnostic imaging*; D005260:Female; D006801:Humans; D007091:Image Processing, Computer-Assisted / Q000379:methods*; D008327:Mammography / Q000379:methods*; D015588:Observer Variation; D012372:ROC Curve; D012016:Reference Values