ROC analysis in medical imaging: a tutorial review of the literature.
Receiver operating characteristic (ROC) analysis measures the "diagnostic accuracy" of a medical imaging system, which represents the second level of diagnostic efficacy in the hierarchical model described by Fryback and Thornbury (Med Decis Making 11:88-94, 1991). After describing the historical origins of ROC analysis, this paper reviews the importance of sampling cases appropriately, designing an observer study to avoid bias, and collecting data on a useful scale. A variety of methods for fitting ROC curves to observer data and testing the statistical significance of apparent differences are then reported. Finally, generalized forms of ROC analysis that require lesion localization or allow more than two states of truth are surveyed briefly.
Metz
Charles E
CE
Radiology and Medical Physics, The University of Chicago, Chicago, IL, USA. c-metz@uchicago.edu
D003952:Diagnostic Imaging / Q000379:methods* / Q000706:statistics & numerical data*; D012372:ROC Curve; D015203:Reproducibility of Results; D012984:Software
Acad Radiol. 2007 Jan;14(1):4-18
17178361
Acad Radiol. 2004 Nov;11(11):1260-73
15561573
Invest Radiol. 1973 Sep-Oct;8(5):339-44
4744434
Stat Med. 1997 Mar 30;16(6):669-79
9131755
Acad Radiol. 2007 Jun;14(6):641-2
17502252
Radiology. 1996 Jun;199(3):843-8
8638015
Acad Radiol. 2006 Oct;13(10):1187-93
16979067
Med Phys. 2002 Dec;29(12):2861-70
12512721
Semin Nucl Med. 1978 Oct;8(4):299-306
112682
Phys Med Biol. 1977 Mar;22(2):376-81
854532
Am J Clin Pathol. 1983 Jan;79(1):78-86
6849296
Stat Med. 1989 Oct;8(10):1277-90
2814075
Am J Roentgenol Radium Ther Nucl Med. 1973 Apr;117(4):757-64
4698817
Radiology. 1996 Dec;201(3):745-50
8939225
Acad Radiol. 2007 Jan;14(1):38-48
17178364
Acad Radiol. 2007 Jul;14(7):814-29
17574132
Med Phys. 1989 Jul-Aug;16(4):561-8
2770630
Med Decis Making. 1988 Jul-Sep;8(3):204-15
3294553
Acad Radiol. 1997 Apr;4(4):298-303
9110028
J Math Psychol. 1999 Mar;43(1):1-33
10069933
Biometrics. 1983 Mar;39(1):207-15
6871349
J Am Coll Radiol. 2006 Jun;3(6):413-22
17412096
Invest Radiol. 1993 Oct;28(10):962-3
8262752
Radiology. 1988 May;167(2):565-9
3357976
Acad Radiol. 2002 Feb;9(2):147-56
11918367
Med Phys. 2004 Jan;31(1):81-90
14761024
N Engl J Med. 1975 Jul 31;293(5):211-5
806804
Med Decis Making. 1991 Apr-Jun;11(2):88-94
1907710
Radiology. 1972 Oct;105(1):199-200
5057304
Semin Nucl Med. 1978 Oct;8(4):283-98
112681
N Engl J Med. 1978 Oct 26;299(17):926-30
692598
Radiology. 1983 Sep;148(3):839-43
6878708
IEEE Trans Med Imaging. 2007 Jan;26(1):77-83
17243586
Acad Radiol. 1995 Mar;2 Suppl 1:S22-9; discussion S57-64, S70-1 pas
9419702
Invest Radiol. 1974 May-Jun;9(3):166-73
4836711
IEEE Trans Med Imaging. 2005 Dec;24(12):1566-73
16350917
Acad Radiol. 1998 Sep;5(9):591-602
9750888
Chest. 1975 Apr;67(4):378-9
1122763
Psychol Rev. 1954 Nov;61(6):401-9
13215690
Crit Rev Diagn Imaging. 1989;29(3):307-35
2667567
Invest Radiol. 1992 Feb;27(2):169-72
1601610
Acad Radiol. 2007 Jun;14(6):723-48
17502262
Ann Intern Med. 1981 Apr;94(4 Pt 2):557-92
6452080
Acad Radiol. 2000 Jul;7(7):516-25
10902960
Psychol Rev. 1961 Sep;68:301-40
13774292
Stat Med. 2005 May 30;24(10):1579-607
15685718
Invest Radiol. 1979 Mar-Apr;14(2):109-21
478799
Radiology. 1975 Sep;116(3):533-8
1153755
Stat Med. 1996 Aug 15;15(15):1687-93
8858790
IEEE Trans Med Imaging. 2008 Feb;27(2):215-27
18334443
Stat Med. 1998 May 15;17(9):1033-53
9612889
Radiology. 1990 Mar;174(3 Pt 1):873-81
2305073
Acad Radiol. 2001 Jul;8(7):605-15
11450961
Med Decis Making. 1984;4(2):151-64
6472063
Acad Radiol. 2000 May;7(5):341-9
10803614
Clin Chem. 1993 Apr;39(4):561-77
8472349
IEEE Trans Med Imaging. 2005 Mar;24(3):293-9
15754980
Radiology. 2003 Jul;228(1):10-4
12832568
N Engl J Med. 1971 Feb 25;284(8):416-24
4395661
Acad Radiol. 2007 Jan;14(1):49-53
17178365
Invest Radiol. 1993 Jan;28(1):59-63
8425854
Med Decis Making. 1997 Jan-Mar;17(1):94-102
8994156
Science. 1971 Mar 26;171(3977):1217-9
5545199
Med Decis Making. 1989 Jul-Sep;9(3):190-5
2668680
Stat Med. 1996 Jul 30;15(14):1575-85
8855483
Acad Radiol. 2004 Sep;11(9):980-95
15350579
Acad Radiol. 1995 Jul;2(7):626-33
9419615
J Nucl Med. 1978 Feb;19(2):213-20
627904
Acad Radiol. 2001 Jul;8(7):616-22
11450962
Invest Radiol. 1986 Sep;21(9):720-33
3095258
Radiology. 1982 Apr;143(1):29-36
7063747
Radiology. 1974 Jan;110(1):89-95
4808546
Phys Med Biol. 2006 Jul 21;51(14):3449-62
16825742
Stat Med. 1996 Aug 30;15(16):1807-26
8870162
IEEE Trans Med Imaging. 2004 Jul;23(7):891-5
15250641
Med Decis Making. 1998 Jan-Mar;18(1):110-21
9456215
Invest Radiol. 1989 Mar;24(3):234-45
2753640
Med Phys. 2004 Aug;31(8):2313-30
15377098
Med Phys. 1996 Oct;23(10):1709-25
8946368
Acad Radiol. 1997 May;4(5):380-9
9156236
Biometrics. 1999 Jun;55(2):488-96
11318205
IEEE Trans Med Imaging. 2006 May;25(5):571-81
16689261
Science. 1973 Dec 7;182(4116):990-1000
17833780
Acad Radiol. 2005 Dec;12(12):1534-41
16321742
Med Decis Making. 1988 Jul-Sep;8(3):197-203
3398748
Invest Radiol. 1983 Mar-Apr;18(2):194-8
6862810
Invest Radiol. 1992 Sep;27(9):723-31
1399456
Med Phys. 1995 Jul;22(7):1057-61
7565380
Psychol Bull. 1986 Mar;99(2):181-98
3515382
Med Decis Making. 1998 Oct-Dec;18(4):436-43
10372587
Acad Radiol. 1997 Feb;4(2):138-49
9061087
Radiology. 1960 Feb;74:178-93
14419034
Biometrics. 1988 Sep;44(3):837-45
3203132
AJR Am J Roentgenol. 2000 Sep;175(3):603-8
10954438
Invest Radiol. 1990 Mar;25(3):230-4
2332308
Acad Radiol. 2001 Apr;8(4):328-34
11293781
Acad Radiol. 1995 Oct;2(10):907-15
9419659
Science. 1988 Jun 3;240(4857):1285-93
3287615
Acad Radiol. 1996 Mar;3(3):245-53
8796672
Acad Radiol. 2002 Nov;9(11):1264-77
12449359
Acad Radiol. 1997 Aug;4(8):587-600
9261459