Meta-interpretive reliability of computer-based test interpretations: the Karson Clinical Report.
Meta-interpretive reliability is a new method to evaluate the accuracy with which personality trait scores are communicated via interpretive statements in a computer-based test interpretation (CBTI). The prototypic experimental design is based on a two-way repeated measures analysis of variance (ANOVA); the two effects are personality traits and randomly chosen CBTI protocols. In this application, 101 psychologists read four examples of the Karson Clinical Report (KCR, Karson & O'Dell, 1975) and estimated the original trait scores from the Sixteen Personality Factor Questionnaire (16PF; Cattell, Eber, & Tatsuoka, 1970) on which the KCR is based. Estimated trait score variance was significantly related to the Trait x Protocol interaction and the main effects for personality trait and differences among protocols (omega 2 = .55). The total effect size corresponded to a multiple correlation of .74, suggesting that the KCR had acceptable meta-interpretive reliability. The protocol effect denoted a context effect created by the juxtaposition of several interpretive statements. Additional analyses showed that individual differences among raters contributed to less than 1% of the estimated standard ten (sten) score variance. Meta-interpretive reliability is proposed as an index of the upper limit of validity for CBTIs.
Endres
L S
LS
Johnson Controls, Inc., Milwaukee, WI.
Guastello
S J
SJ
Rieke
M L
ML
D000328:Adult; D000368:Aged; D003142:Communication; D005260:Female; D006801:Humans; D008297:Male; D008875:Middle Aged; D015233:Models, Statistical; D010551:Personality; D010552:Personality Assessment; D015203:Reproducibility of Results; D012107:Research Design