title: Reliability of functional magnetic resonance imaging activation during working memory in a multi-site study: analysis from the North American Prodrome Longitudinal Study.
abstract: Multi-site neuroimaging studies offer an efficient means to study brain functioning in large samples of individuals with rare conditions; however, they present new challenges given that aggregating data across sites introduces additional variability into measures of interest. Assessing the reliability of brain activation across study sites and comparing statistical methods for pooling functional data are critical to ensuring the validity of aggregating data across sites. The current study used two samples of healthy individuals to assess the feasibility and reliability of aggregating multi-site functional magnetic resonance imaging (fMRI) data from a Sternberg-style verbal working memory task. Participants were recruited as part of the North American Prodrome Longitudinal Study (NAPLS), which comprises eight fMRI scanning sites across the United States and Canada. In the first study sample (n=8), one participant from each home site traveled to each of the sites and was scanned while completing the task on two consecutive days. Reliability was examined using generalizability theory. Results indicated that blood oxygen level-dependent (BOLD) signal was reproducible across sites and was highly reliable, or generalizable, across scanning sites and testing days for core working memory ROIs (generalizability ICCs=0.81 for left dorsolateral prefrontal cortex, 0.95 for left superior parietal cortex). In the second study sample (n=154), two statistical methods for aggregating fMRI data across sites for all healthy individuals recruited as control participants in the NAPLS study were compared. Control participants were scanned on one occasion at the site from which they were recruited. Results from the image-based meta-analysis (IBMA) method and mixed effects model with site covariance method both showed robust activation in expected regions (i.e. dorsolateral prefrontal cortex, anterior cingulate cortex, supplementary motor cortex, superior parietal cortex, inferior temporal cortex, cerebellum, thalamus, basal ganglia). Quantification of the similarity of group maps from these methods confirmed a very high (96%) degree of spatial overlap in results. Thus, brain activation during working memory function was reliable across the NAPLS sites and both the IBMA and mixed effects model with site covariance methods appear to be valid approaches for aggregating data across sites. These findings indicate that multi-site functional neuroimaging can offer a reliable means to increase power and generalizability of results when investigating brain function in rare populations and support the multi-site investigation of working memory function in the NAPLS study, in particular.
authors.affiliation: University of California, Los Angeles, Los Angeles, CA, United States.
authors.forename: Jennifer K
authors.initials: JK
authors.lastname: Forsyth
authors.affiliation: University of California, Los Angeles, Los Angeles, CA, United States.
authors.forename: Sarah C
authors.initials: SC
authors.lastname: McEwen
authors.affiliation: University of California, Los Angeles, Los Angeles, CA, United States.
authors.forename: Dylan G
authors.initials: DG
authors.lastname: Gee
authors.affiliation: University of California, Los Angeles, Los Angeles, CA, United States.
authors.forename: Carrie E
authors.initials: CE
authors.lastname: Bearden
authors.affiliation: University of Calgary, Calgary, Alberta, Canada.
authors.forename: Jean
authors.initials: J
authors.lastname: Addington
authors.affiliation: University of Calgary, Calgary, Alberta, Canada.
authors.forename: Brad
authors.initials: B
authors.lastname: Goodyear
authors.affiliation: University of California, San Diego, San Diego, CA, United States.
authors.forename: Kristin S
authors.initials: KS
authors.lastname: Cadenhead
authors.affiliation: University of California, San Diego, San Diego, CA, United States.
authors.forename: Heline
authors.initials: H
authors.lastname: Mirzakhanian
authors.affiliation: Zucker Hillside Hospital, Great Neck, NY, United States.
authors.forename: Barbara A
authors.initials: BA
authors.lastname: Cornblatt
authors.affiliation: Zucker Hillside Hospital, Great Neck, NY, United States.
authors.forename: Doreen M
authors.initials: DM
authors.lastname: Olvet
authors.affiliation: University of California, San Francisco, San Francisco, CA, United States.
authors.forename: Daniel H
authors.initials: DH
authors.lastname: Mathalon
authors.affiliation: Yale University, New Haven, CT, United States.
authors.forename: Thomas H
authors.initials: TH
authors.lastname: McGlashan
authors.affiliation: University of North Carolina, Chapel Hill, Chapel Hill, NC, United States.
authors.forename: Diana O
authors.initials: DO
authors.lastname: Perkins
authors.affiliation: University of North Carolina, Chapel Hill, Chapel Hill, NC, United States.
authors.forename: Aysenil
authors.initials: A
authors.lastname: Belger
authors.affiliation: Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, MA, United States.
authors.forename: Larry J
authors.initials: LJ
authors.lastname: Seidman
authors.affiliation: Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, MA, United States.
authors.forename: Heidi W
authors.initials: HW
authors.lastname: Thermenos
authors.affiliation: University of California, San Diego, San Diego, CA, United States.
authors.forename: Ming T
authors.initials: MT
authors.lastname: Tsuang
authors.affiliation: University of California, Irvine, Irvine, CA, United States.
authors.forename: Theo G M
authors.initials: TG
authors.lastname: van Erp
authors.affiliation: Emory University, Atlanta, GA, United States.
authors.forename: Elaine F
authors.initials: EF
authors.lastname: Walker
authors.affiliation: Emory University, Atlanta, GA, United States.
authors.forename: Stephan
authors.initials: S
authors.lastname: Hamann
authors.affiliation: Yale University, New Haven, CT, United States.
authors.forename: Scott W
authors.initials: SW
authors.lastname: Woods
authors.affiliation: Yale University, New Haven, CT, United States.
authors.forename: Maolin
authors.initials: M
authors.lastname: Qiu
authors.affiliation: Yale University, New Haven, CT, United States. Electronic address: tyrone.cannon@yale.edu.
authors.forename: Tyrone D
authors.initials: TD
authors.lastname: Cannon
mesh_terms: D000293:Adolescent; D000328:Adult; D001921:Brain / Q000473:pathology; D002170:Canada; D005260:Female; D006801:Humans; D007091:Image Processing, Computer-Assisted; D008137:Longitudinal Studies; D008279:Magnetic Resonance Imaging / Q000379:methods*; D008297:Male; D008570:Memory, Short-Term / Q000502:physiology*; D015337:Multicenter Studies as Topic / Q000379:methods*; D010100:Oxygen / Q000097:blood; D011597:Psychomotor Performance / Q000502:physiology; D011930:Reaction Time / Q000502:physiology; D015203:Reproducibility of Results; D014481:United States; D055815:Young Adult
chemical_list: D010100:Oxygen
keywords: G-theory; Multi-site; Reliability; Working memory; fMRI
references.citation: Neuroimage. 2003 Feb;18(2):293-309
references.pmid: 12595184
references.citation: Proc Natl Acad Sci U S A. 2010 Oct 19;107(42):18167-72
references.pmid: 20921401
references.citation: Hum Brain Mapp. 2008 Oct;29(10):1111-22
references.pmid: 17680602
references.citation: Neuroimage. 2010 Oct 15;53(1):119-31
references.pmid: 20451631
references.citation: Am J Ment Defic. 1981 Sep;86(2):127-37
references.pmid: 7315877
references.citation: Radiology. 2005 Dec;237(3):781-9
references.pmid: 16304101
references.citation: Hum Brain Mapp. 2008 Aug;29(8):958-72
references.pmid: 17636563
references.citation: J Cereb Blood Flow Metab. 1992 Nov;12(6):900-18
references.pmid: 1400644
references.citation: Med Image Anal. 2001 Jun;5(2):143-56
references.pmid: 11516708
references.citation: Psychiatry Res. 2010 Nov 30;184(2):86-95
references.pmid: 20880670
references.citation: Neuroimage. 2004;23 Suppl 1:S208-19
references.pmid: 15501092
references.citation: Neuroimage. 2011 Feb 1;54(3):2163-75
references.pmid: 20932915
references.citation: Biol Psychiatry. 2011 Jul 1;70(1):13-8
references.pmid: 21334602
references.citation: Nat Neurosci. 2003 Jul;6(7):750-7
references.pmid: 12808459
references.citation: Front Syst Neurosci. 2012 Aug 16;6:59
references.pmid: 22912605
references.citation: Magn Reson Med. 1995 May;33(5):636-47
references.pmid: 7596267
references.citation: Neuroimage. 2010 Jan 1;49(1):552-60
references.pmid: 19631757
references.citation: Front Neuroinform. 2009 Sep 30;3:33
references.pmid: 19826498
references.citation: Prog Neuropsychopharmacol Biol Psychiatry. 2011 Mar 30;35(2):473-82
references.pmid: 21185903
references.citation: J Abnorm Psychol. 2005 Nov;114(4):599-611
references.pmid: 16351383
references.citation: Neuroimage. 2009 Apr 15;45(3):758-68
references.pmid: 19166942
references.citation: Psychol Bull. 1979 Mar;86(2):420-8
references.pmid: 18839484
references.citation: Ann N Y Acad Sci. 2010 Mar;1191:133-55
references.pmid: 20392279
references.citation: Neuroimage. 2003 Jul;19(3):1233-9
references.pmid: 12880848
references.citation: Neuroimage. 2009 Apr 15;45(3):810-23
references.pmid: 19166944
references.citation: Hum Brain Mapp. 2009 Mar;30(3):859-73
references.pmid: 18330870
references.citation: Am J Psychiatry. 2010 Feb;167(2):160-9
references.pmid: 20048021
references.citation: Neuroimage. 2001 Dec;14(6):1370-86
references.pmid: 11707093
references.citation: Neuroimage. 2013 Feb 1;66:648-61
references.pmid: 23153970
references.citation: Acad Radiol. 2008 Sep;15(9):1099-110
references.pmid: 18692750
references.citation: Neuroimage. 2002 Oct;17(2):825-41
references.pmid: 12377157
references.citation: Neuron. 2006 Jul 20;51(2):251-61
references.pmid: 16846859
references.citation: Am J Psychiatry. 2003 Oct;160(10):1809-16
references.pmid: 14514495
references.citation: J Magn Reson Imaging. 2012 Jul;36(1):39-54
references.pmid: 22314879
references.citation: Science. 1966 Aug 5;153(3736):652-4
references.pmid: 5939936
references.citation: Cereb Cortex. 2010 Jan;20(1):46-60
references.pmid: 19395526
references.citation: Schizophr Bull. 2011 Jan;37(1):222-32
references.pmid: 19770491
references.citation: Neuropsychologia. 2005;43(6):930-42
references.pmid: 15716163
references.citation: Arch Gen Psychiatry. 2012 Jun;69(6):562-71
references.pmid: 22664547