title: ROC analysis in medical imaging: a tutorial review of the literature.
abstract: Receiver operating characteristic (ROC) analysis measures the "diagnostic accuracy" of a medical imaging system, which represents the second level of diagnostic efficacy in the hierarchical model described by Fryback and Thornbury (Med Decis Making 11:88-94, 1991). After describing the historical origins of ROC analysis, this paper reviews the importance of sampling cases appropriately, designing an observer study to avoid bias, and collecting data on a useful scale. A variety of methods for fitting ROC curves to observer data and testing the statistical significance of apparent differences are then reported. Finally, generalized forms of ROC analysis that require lesion localization or allow more than two states of truth are surveyed briefly.
authors.affiliation: Radiology and Medical Physics, The University of Chicago, Chicago, IL, USA. c-metz@uchicago.edu
authors.forename: Charles E
authors.initials: CE
authors.lastname: Metz
mesh_terms: D003952:Diagnostic Imaging / Q000379:methods* / Q000706:statistics & numerical data*; D012372:ROC Curve; D015203:Reproducibility of Results; D012984:Software
references.citation: Acad Radiol. 2007 Jan;14(1):4-18
references.pmid: 17178361
references.citation: Acad Radiol. 2004 Nov;11(11):1260-73
references.pmid: 15561573
references.citation: Invest Radiol. 1973 Sep-Oct;8(5):339-44
references.pmid: 4744434
references.citation: Stat Med. 1997 Mar 30;16(6):669-79
references.pmid: 9131755
references.citation: Acad Radiol. 2007 Jun;14(6):641-2
references.pmid: 17502252
references.citation: Radiology. 1996 Jun;199(3):843-8
references.pmid: 8638015
references.citation: Acad Radiol. 2006 Oct;13(10):1187-93
references.pmid: 16979067
references.citation: Med Phys. 2002 Dec;29(12):2861-70
references.pmid: 12512721
references.citation: Semin Nucl Med. 1978 Oct;8(4):299-306
references.pmid: 112682
references.citation: Phys Med Biol. 1977 Mar;22(2):376-81
references.pmid: 854532
references.citation: Am J Clin Pathol. 1983 Jan;79(1):78-86
references.pmid: 6849296
references.citation: Stat Med. 1989 Oct;8(10):1277-90
references.pmid: 2814075
references.citation: Am J Roentgenol Radium Ther Nucl Med. 1973 Apr;117(4):757-64
references.pmid: 4698817
references.citation: Radiology. 1996 Dec;201(3):745-50
references.pmid: 8939225
references.citation: Acad Radiol. 2007 Jan;14(1):38-48
references.pmid: 17178364
references.citation: Acad Radiol. 2007 Jul;14(7):814-29
references.pmid: 17574132
references.citation: Med Phys. 1989 Jul-Aug;16(4):561-8
references.pmid: 2770630
references.citation: Med Decis Making. 1988 Jul-Sep;8(3):204-15
references.pmid: 3294553
references.citation: Acad Radiol. 1997 Apr;4(4):298-303
references.pmid: 9110028
references.citation: J Math Psychol. 1999 Mar;43(1):1-33
references.pmid: 10069933
references.citation: Biometrics. 1983 Mar;39(1):207-15
references.pmid: 6871349
references.citation: J Am Coll Radiol. 2006 Jun;3(6):413-22
references.pmid: 17412096
references.citation: Invest Radiol. 1993 Oct;28(10):962-3
references.pmid: 8262752
references.citation: Radiology. 1988 May;167(2):565-9
references.pmid: 3357976
references.citation: Acad Radiol. 2002 Feb;9(2):147-56
references.pmid: 11918367
references.citation: Med Phys. 2004 Jan;31(1):81-90
references.pmid: 14761024
references.citation: N Engl J Med. 1975 Jul 31;293(5):211-5
references.pmid: 806804
references.citation: Med Decis Making. 1991 Apr-Jun;11(2):88-94
references.pmid: 1907710
references.citation: Radiology. 1972 Oct;105(1):199-200
references.pmid: 5057304
references.citation: Semin Nucl Med. 1978 Oct;8(4):283-98
references.pmid: 112681
references.citation: N Engl J Med. 1978 Oct 26;299(17):926-30
references.pmid: 692598
references.citation: Radiology. 1983 Sep;148(3):839-43
references.pmid: 6878708
references.citation: IEEE Trans Med Imaging. 2007 Jan;26(1):77-83
references.pmid: 17243586
references.citation: Acad Radiol. 1995 Mar;2 Suppl 1:S22-9; discussion S57-64, S70-1 pas
references.pmid: 9419702
references.citation: Invest Radiol. 1974 May-Jun;9(3):166-73
references.pmid: 4836711
references.citation: IEEE Trans Med Imaging. 2005 Dec;24(12):1566-73
references.pmid: 16350917
references.citation: Acad Radiol. 1998 Sep;5(9):591-602
references.pmid: 9750888
references.citation: Chest. 1975 Apr;67(4):378-9
references.pmid: 1122763
references.citation: Psychol Rev. 1954 Nov;61(6):401-9
references.pmid: 13215690
references.citation: Crit Rev Diagn Imaging. 1989;29(3):307-35
references.pmid: 2667567
references.citation: Invest Radiol. 1992 Feb;27(2):169-72
references.pmid: 1601610
references.citation: Acad Radiol. 2007 Jun;14(6):723-48
references.pmid: 17502262
references.citation: Ann Intern Med. 1981 Apr;94(4 Pt 2):557-92
references.pmid: 6452080
references.citation: Acad Radiol. 2000 Jul;7(7):516-25
references.pmid: 10902960
references.citation: Psychol Rev. 1961 Sep;68:301-40
references.pmid: 13774292
references.citation: Stat Med. 2005 May 30;24(10):1579-607
references.pmid: 15685718
references.citation: Invest Radiol. 1979 Mar-Apr;14(2):109-21
references.pmid: 478799
references.citation: Radiology. 1975 Sep;116(3):533-8
references.pmid: 1153755
references.citation: Stat Med. 1996 Aug 15;15(15):1687-93
references.pmid: 8858790
references.citation: IEEE Trans Med Imaging. 2008 Feb;27(2):215-27
references.pmid: 18334443
references.citation: Stat Med. 1998 May 15;17(9):1033-53
references.pmid: 9612889
references.citation: Radiology. 1990 Mar;174(3 Pt 1):873-81
references.pmid: 2305073
references.citation: Acad Radiol. 2001 Jul;8(7):605-15
references.pmid: 11450961
references.citation: Med Decis Making. 1984;4(2):151-64
references.pmid: 6472063
references.citation: Acad Radiol. 2000 May;7(5):341-9
references.pmid: 10803614
references.citation: Clin Chem. 1993 Apr;39(4):561-77
references.pmid: 8472349
references.citation: IEEE Trans Med Imaging. 2005 Mar;24(3):293-9
references.pmid: 15754980
references.citation: Radiology. 2003 Jul;228(1):10-4
references.pmid: 12832568
references.citation: N Engl J Med. 1971 Feb 25;284(8):416-24
references.pmid: 4395661
references.citation: Acad Radiol. 2007 Jan;14(1):49-53
references.pmid: 17178365
references.citation: Invest Radiol. 1993 Jan;28(1):59-63
references.pmid: 8425854
references.citation: Med Decis Making. 1997 Jan-Mar;17(1):94-102
references.pmid: 8994156
references.citation: Science. 1971 Mar 26;171(3977):1217-9
references.pmid: 5545199
references.citation: Med Decis Making. 1989 Jul-Sep;9(3):190-5
references.pmid: 2668680
references.citation: Stat Med. 1996 Jul 30;15(14):1575-85
references.pmid: 8855483
references.citation: Acad Radiol. 2004 Sep;11(9):980-95
references.pmid: 15350579
references.citation: Acad Radiol. 1995 Jul;2(7):626-33
references.pmid: 9419615
references.citation: J Nucl Med. 1978 Feb;19(2):213-20
references.pmid: 627904
references.citation: Acad Radiol. 2001 Jul;8(7):616-22
references.pmid: 11450962
references.citation: Invest Radiol. 1986 Sep;21(9):720-33
references.pmid: 3095258
references.citation: Radiology. 1982 Apr;143(1):29-36
references.pmid: 7063747
references.citation: Radiology. 1974 Jan;110(1):89-95
references.pmid: 4808546
references.citation: Phys Med Biol. 2006 Jul 21;51(14):3449-62
references.pmid: 16825742
references.citation: Stat Med. 1996 Aug 30;15(16):1807-26
references.pmid: 8870162
references.citation: IEEE Trans Med Imaging. 2004 Jul;23(7):891-5
references.pmid: 15250641
references.citation: Med Decis Making. 1998 Jan-Mar;18(1):110-21
references.pmid: 9456215
references.citation: Invest Radiol. 1989 Mar;24(3):234-45
references.pmid: 2753640
references.citation: Med Phys. 2004 Aug;31(8):2313-30
references.pmid: 15377098
references.citation: Med Phys. 1996 Oct;23(10):1709-25
references.pmid: 8946368
references.citation: Acad Radiol. 1997 May;4(5):380-9
references.pmid: 9156236
references.citation: Biometrics. 1999 Jun;55(2):488-96
references.pmid: 11318205
references.citation: IEEE Trans Med Imaging. 2006 May;25(5):571-81
references.pmid: 16689261
references.citation: Science. 1973 Dec 7;182(4116):990-1000
references.pmid: 17833780
references.citation: Acad Radiol. 2005 Dec;12(12):1534-41
references.pmid: 16321742
references.citation: Med Decis Making. 1988 Jul-Sep;8(3):197-203
references.pmid: 3398748
references.citation: Invest Radiol. 1983 Mar-Apr;18(2):194-8
references.pmid: 6862810
references.citation: Invest Radiol. 1992 Sep;27(9):723-31
references.pmid: 1399456
references.citation: Med Phys. 1995 Jul;22(7):1057-61
references.pmid: 7565380
references.citation: Psychol Bull. 1986 Mar;99(2):181-98
references.pmid: 3515382
references.citation: Med Decis Making. 1998 Oct-Dec;18(4):436-43
references.pmid: 10372587
references.citation: Acad Radiol. 1997 Feb;4(2):138-49
references.pmid: 9061087
references.citation: Radiology. 1960 Feb;74:178-93
references.pmid: 14419034
references.citation: Biometrics. 1988 Sep;44(3):837-45
references.pmid: 3203132
references.citation: AJR Am J Roentgenol. 2000 Sep;175(3):603-8
references.pmid: 10954438
references.citation: Invest Radiol. 1990 Mar;25(3):230-4
references.pmid: 2332308
references.citation: Acad Radiol. 2001 Apr;8(4):328-34
references.pmid: 11293781
references.citation: Acad Radiol. 1995 Oct;2(10):907-15
references.pmid: 9419659
references.citation: Science. 1988 Jun 3;240(4857):1285-93
references.pmid: 3287615
references.citation: Acad Radiol. 1996 Mar;3(3):245-53
references.pmid: 8796672
references.citation: Acad Radiol. 2002 Nov;9(11):1264-77
references.pmid: 12449359
references.citation: Acad Radiol. 1997 Aug;4(8):587-600
references.pmid: 9261459