title: Estimating sensitivity and specificity for technology assessment based on observer studies.
abstract: OBJECTIVE
  The goal of this study was to determine the accuracy and precision of using scores from a receiver operating characteristic rating scale to estimate sensitivity and specificity.
  
  
  METHODS
  We used data collected in a previous study that measured the improvements in radiologists' ability to classify mammographic microcalcification clusters as benign or malignant with and without the use of a computer-aided diagnosis scheme. Sensitivity and specificity were estimated from the rating data from a question that directly asked the radiologists their biopsy recommendations, which was used as the "truth," because it is the actual recall decision, thus it is their subjective truth. By thresholding the rating data, sensitivity and specificity were estimated for different threshold values.
  
  
  RESULTS
  Because of interreader and intrareader variability, estimated sensitivity and specificity values for individual readers could be as much as 100% in error when using rating data compared to using the biopsy recommendation data. When pooled together, the estimates using thresholding the rating data were in good agreement with sensitivity and specificity estimated from the recommendation data. However, the statistical power of the rating data estimates was lower.
  
  
  CONCLUSIONS
  By simply asking the observer his or her explicit recommendation (eg, biopsy or no biopsy), sensitivity and specificity can be measured directly, giving a more accurate description of empirical variability and the power of the study can be maximized.
authors: Department of Radiology, The University of Chicago, 5841 South Maryland Avenue, MC-2026, Chicago, IL 60637, USA. r-nishikawa@uchicago.edu
authors: Robert M
authors: RM
authors: Nishikawa
authors: Lorenzo L
authors: LL
authors: Pesce
mesh_terms: D001706:Biopsy; D001941:Breast Diseases / Q000000981:diagnostic imaging; D001943:Breast Neoplasms / Q000000981:diagnostic imaging*; D003936:Diagnosis, Computer-Assisted / Q000379:methods / Q000592:standards; D003937:Diagnosis, Differential; D005260:Female; D006801:Humans; D008327:Mammography / Q000379:methods / Q000592:standards*; D015588:Observer Variation; D012372:ROC Curve; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D013673:Technology Assessment, Biomedical / Q000379:methods* / Q000706:statistics & numerical data