title: AGREEMENT AND COVERAGE OF INDICATORS OF RESPONSE TO INTERVENTION: A MULTI-METHOD COMPARISON AND SIMULATION.
abstract: OBJECTIVE
  Agreement across methods for identifying students as inadequate responders or as learning disabled is often poor. We report (1) an empirical examination of final status (post-intervention benchmarks) and dual-discrepancy growth methods based on growth during the intervention and final status for assessing response to intervention; and (2) a statistical simulation of psychometric issues that may explain low agreement.
  
  
  METHODS
  After a Tier 2 intervention, final status benchmark criteria were used to identify 104 inadequate and 85 adequate responders to intervention, with comparisons of agreement and coverage for these methods and a dual-discrepancy method. Factors affecting agreement were investigated using computer simulation to manipulate reliability, the intercorrelation between measures, cut points, normative samples, and sample size.
  
  
  RESULTS
  Identification of inadequate responders based on individual measures showed that single measures tended not to identify many members of the pool of 104 inadequate responders. Poor to fair levels of agreement for identifying inadequate responders were apparent between pairs of measures In the simulation, comparisons across two simulated measures generated indices of agreement (kappa) that were generally low because of multiple psychometric issues inherent in any test.
  
  
  CONCLUSIONS
  Expecting excellent agreement between two correlated tests with even small amounts of unreliability may not be realistic. Assessing outcomes based on multiple measures, such as level of CBM performance and short norm-referenced assessments of fluency may improve the reliability of diagnostic decisions.
authors.affiliation: Texas Institute of Measurement, Evaluation, and Statistics and Department of Psychology, University of Houston.
authors.forename: Jack M
authors.initials: JM
authors.lastname: Fletcher
authors.affiliation: Texas Institute of Measurement, Evaluation, and Statistics and Department of Psychology, University of Houston.
authors.forename: Karla K
authors.initials: KK
authors.lastname: Stuebing
authors.affiliation: Texas Institute of Measurement, Evaluation, and Statistics and Department of Psychology, University of Houston.
authors.forename: Amy E
authors.initials: AE
authors.lastname: Barth
authors.affiliation: Texas Institute of Measurement, Evaluation, and Statistics and Department of Psychology, University of Houston.
authors.forename: Jeremy
authors.initials: J
authors.lastname: Miciak
authors.affiliation: Texas Institute of Measurement, Evaluation, and Statistics and Department of Psychology, University of Houston.
authors.forename: David J
authors.initials: DJ
authors.lastname: Francis
authors.affiliation: Children's Learning Institute, University of Texas Health Science Center at Houston.
authors.forename: Carolyn A
authors.initials: CA
authors.lastname: Denton
keywords: decision reliability; diagnostic agreement; kappa; learning disabilities; response to intervention; simulation