Does assessment type matter? A measurement invariance analysis of online and paper and pencil assessment of the Community Assessment of Psychic Experiences (CAPE).
BACKGROUND
The psychometric properties of an online test are not necessarily identical to its paper and pencil original. The aim of this study is to test whether the factor structure of the Community Assessment of Psychic Experiences (CAPE) is measurement invariant with respect to online vs. paper and pencil assessment.


METHODS
The factor structure of CAPE items assessed by paper and pencil (N = 796) was compared with the factor structure of CAPE items assessed by the Internet (N = 21,590) using formal tests for Measurement Invariance (MI). The effect size was calculated by estimating the Signed Item Difference in the Sample (SIDS) index and the Signed Test Difference in the Sample (STDS) for a hypothetical subject who scores 2 standard deviations above average on the latent dimensions.


RESULTS
The more restricted Metric Invariance model showed a significantly worse fit compared to the less restricted Configural Invariance model (χ(2)(23) = 152.75, p<0.001). However, the SIDS indices appear to be small, with an average of -0.11. A STDS of -4.80 indicates that Internet sample members who score 2 standard deviations above average would be expected to score 4.80 points lower on the CAPE total scale (ranging from 42 to 114 points) than would members of the Paper sample with the same latent trait score.


CONCLUSIONS
Our findings did not support measurement invariance with respect to assessment method. Because of the small effect sizes, the measurement differences between the online assessed CAPE and its paper and pencil original can be neglected without major consequences for research purposes. However, a person with a high vulnerability for psychotic symptoms would score 4.80 points lower on the total scale if the CAPE is assessed online compared to paper and pencil assessment. Therefore, for clinical purposes, one should be cautious with online assessment of the CAPE.
Brain Center Rudolf Magnus, Department of Psychiatry, University Medical Center Utrecht, Utrecht, The Netherlands.
Marloes
M
Vleeschouwer
Brain Center Rudolf Magnus, Department of Psychiatry, University Medical Center Utrecht, Utrecht, The Netherlands.
Chris D
CD
Schubart
Department of Psychiatry and Psychology, School of Mental Health and Neuroscience, Maastricht University Medical Centre, Maastricht, The Netherlands.
Cecile
C
Henquet
Department of Psychiatry and Psychology, School of Mental Health and Neuroscience, Maastricht University Medical Centre, Maastricht, The Netherlands.
Inez
I
Myin-Germeys
Brain Center Rudolf Magnus, Department of Psychiatry, University Medical Center Utrecht, Utrecht, The Netherlands.
Willemijn A
WA
van Gastel
Brain Center Rudolf Magnus, Department of Psychiatry, University Medical Center Utrecht, Utrecht, The Netherlands.
Manon H J
MH
Hillegers
Department of Psychiatry and Psychology, School of Mental Health and Neuroscience, Maastricht University Medical Centre, Maastricht, The Netherlands.
Jim J
JJ
van Os
Brain Center Rudolf Magnus, Department of Psychiatry, University Medical Center Utrecht, Utrecht, The Netherlands.
Marco P M
MP
Boks
Brain Center Rudolf Magnus, Department of Psychiatry, University Medical Center Utrecht, Utrecht, The Netherlands ; Department of Psychiatry, Academic Medical Center Amsterdam, Amsterdam, The Netherlands.
Eske M
EM
Derks
D000293:Adolescent; D000328:Adult; D002648:Child; D005260:Female; D006801:Humans; D020407:Internet; D008297:Male; D008875:Middle Aged; D010209:Paper; D011581:Psychological Tests / Q000592:standards*; D011594:Psychometrics / Q000379:methods / Q000592:standards*; D011618:Psychotic Disorders / Q000175:diagnosis / Q000523:psychology; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D011795:Surveys and Questionnaires; D055815:Young Adult