title: Quantifying the reliability of image replication studies: the image intraclass correlation coefficient (I2C2).
abstract: This article proposes the image intraclass correlation (I2C2) coefficient as a global measure of reliability for imaging studies. The I2C2 generalizes the classic intraclass correlation (ICC) coefficient to the case when the data of interest are images, thereby providing a measure that is both intuitive and convenient. Drawing a connection with classical measurement error models for replication experiments, the I2C2 can be computed quickly, even in high-dimensional imaging studies. A nonparametric bootstrap procedure is introduced to quantify the variability of the I2C2 estimator. Furthermore, a Monte Carlo permutation is utilized to test reproducibility versus a zero I2C2, representing complete lack of reproducibility. Methodologies are applied to three replication studies arising from different brain imaging modalities and settings: regional analysis of volumes in normalized space imaging for characterizing brain morphology, seed-voxel brain activation maps based on resting-state functional magnetic resonance imaging (fMRI), and fractional anisotropy in an area surrounding the corpus callosum via diffusion tensor imaging. Notably, resting-state fMRI brain activation maps are found to have low reliability, ranging from .2 to .4. Software and data are available to provide easy access to the proposed methods.
authors.forename: H
authors.initials: H
authors.lastname: Shou
authors.forename: A
authors.initials: A
authors.lastname: Eloyan
authors.forename: S
authors.initials: S
authors.lastname: Lee
authors.forename: V
authors.initials: V
authors.lastname: Zipunnikov
authors.forename: A N
authors.initials: AN
authors.lastname: Crainiceanu
authors.forename: N B
authors.initials: NB
authors.lastname: Nebel
authors.forename: B
authors.initials: B
authors.lastname: Caffo
authors.forename: M A
authors.initials: MA
authors.lastname: Lindquist
authors.forename: C M
authors.initials: CM
authors.lastname: Crainiceanu
mesh_terms: D000328:Adult; D001921:Brain / Q000033:anatomy & histology / Q000473:pathology / Q000502:physiology*; D001931:Brain Mapping; D003198:Computer Simulation; D005260:Female; D006801:Humans; D008297:Male; D008954:Models, Biological; D059906:Neuroimaging / Q000145:classification; D015203:Reproducibility of Results; D013223:Statistics as Topic
references.citation: Brain Res. 2008 Nov 6;1239:141-51
references.pmid: 18789314
references.citation: Neuroimage. 2002 Apr;15(4):747-71
references.pmid: 11906218
references.citation: Neurology. 2011 Jan 11;76(2):179-86
references.pmid: 21220722
references.citation: J Neurophysiol. 2008 Oct;100(4):1800-12
references.pmid: 18684903
references.citation: Neuroimage. 2011 Apr 1;55(3):1132-46
references.pmid: 21194570
references.citation: Proc Natl Acad Sci U S A. 2005 Jul 5;102(27):9673-8
references.pmid: 15976020
references.citation: Ann Appl Stat. 2009 Mar 1;3(1):458-488
references.pmid: 20221415
references.citation: Neuroimage. 2010 Feb 15;49(4):3047-56
references.pmid: 19944769
references.citation: Neuroimage. 2010 Jan 15;49(2):1432-45
references.pmid: 19782143
references.citation: Stat Med. 2012 Nov 20;31(26):3223-40
references.pmid: 22855258
references.citation: Neuroimage. 2011 Jul 15;57(2):431-9
references.pmid: 21554962
references.citation: Electron J Stat. 2010;4:1022-1054
references.pmid: 21743825
references.citation: Mult Scler. 2010 Feb;16(2):166-77
references.pmid: 20142309
references.citation: Cereb Cortex. 2009 Oct;19(10):2209-29
references.pmid: 19221144
references.citation: Proc Natl Acad Sci U S A. 2009 Feb 10;106(6):2035-40
references.pmid: 19188601
references.citation: IEEE Trans Med Imaging. 2002 Nov;21(11):1421-39
references.pmid: 12575879
references.citation: Neuroimage. 2011 Feb 14;54(4):2854-66
references.pmid: 21094686
references.citation: Neuroimage. 2001 Dec;14(6):1361-9
references.pmid: 11707092
references.citation: Neuroscientist. 2006 Apr;12(2):143-52
references.pmid: 16514011
references.citation: Neuroimage. 2010 Feb 1;49(3):2457-66
references.pmid: 19818860
references.citation: Neuroimage. 2007 Aug 1;37(1):90-101
references.pmid: 17560126
references.citation: Magn Reson Imaging. 1998;16(2):105-13
references.pmid: 9508267
references.citation: J Am Stat Assoc. 2009 Dec 1;104(488):1550-1561
references.pmid: 20625442
references.citation: Hum Brain Mapp. 2010 Feb;31(2):237-46
references.pmid: 19621371
references.citation: Ann N Y Acad Sci. 2010 Mar;1191:133-55
references.pmid: 20392279
references.citation: Psychol Bull. 1979 Mar;86(2):420-8
references.pmid: 18839484
references.citation: Proc Natl Acad Sci U S A. 2006 Sep 12;103(37):13848-53
references.pmid: 16945915
references.citation: Neuroimage. 2010 Jan 15;49(2):1524-35
references.pmid: 19766196
references.citation: Neuroimage. 2012 Jan 16;59(2):1404-12
references.pmid: 21888983
references.citation: Neuroimage. 2011 Feb 1;54(3):2033-44
references.pmid: 20851191
references.citation: Neuroimage. 2011 Mar 15;55(2):607-15
references.pmid: 21146616
references.citation: PLoS One. 2011;6(7):e21976
references.pmid: 21818285
references.citation: J Comput Graph Stat. 2011;20(4):852-873
references.pmid: 25960627
references.citation: Neuroimage. 2010 Feb 1;49(3):2163-77
references.pmid: 19896537