title: Prediction of true test scores from observed item scores and ancillary data.
abstract: In many educational tests which involve constructed responses, a traditional test score is obtained by adding together item scores obtained through holistic scoring by trained human raters. For example, this practice was used until 2008 in the case of GRE(®) General Analytical Writing and until 2009 in the case of TOEFL(®) iBT Writing. With use of natural language processing, it is possible to obtain additional information concerning item responses from computer programs such as e-rater(®). In addition, available information relevant to examinee performance may include scores on related tests. We suggest application of standard results from classical test theory to the available data to obtain best linear predictors of true traditional test scores. In performing such analysis, we require estimation of variances and covariances of measurement errors, a task which can be quite difficult in the case of tests with limited numbers of items and with multiple measurements per item. As a consequence, a new estimation method is suggested based on samples of examinees who have taken an assessment more than once. Such samples are typically not random samples of the general population of examinees, so that we apply statistical adjustment methods to obtain the needed estimated variances and covariances of measurement errors. To examine practical implications of the suggested methods of analysis, applications are made to GRE General Analytical Writing and TOEFL iBT Writing. Results obtained indicate that substantial improvements are possible both in terms of reliability of scoring and in terms of assessment reliability.
authors.affiliation: ETS, Princeton, New Jersey, USA.
authors.forename: Shelby J
authors.initials: SJ
authors.lastname: Haberman
authors.forename: Lili
authors.initials: L
authors.lastname: Yao
authors.forename: Sandip
authors.initials: S
authors.lastname: Sinharay
mesh_terms: D000465:Algorithms; D004521:Educational Measurement / Q000706:statistics & numerical data*; D006801:Humans; D008432:Mathematical Computing; D009323:Natural Language Processing; D011932:Reading; D012044:Regression Analysis; D012107:Research Design / Q000706:statistics & numerical data*; D012570:School Admission Criteria / Q000706:statistics & numerical data; D012984:Software; D013060:Speech; D013067:Speech Perception; D014956:Writing
keywords: adjustment; augmentation; automated scoring; classical test theory; holistic scoring; proportional reduction in mean square error; reliability