title: Influences of response rate and distribution on the calculation of interobserver reliability scores.
abstract: We examined the effects of several variations in response rate on the calculation of total, interval, exact-agreement, and proportional reliability indices. Trained observers recorded computer-generated data that appeared on a computer screen. In Study 1, target responses occurred at low, moderate, and high rates during separate sessions so that reliability results based on the four calculations could be compared across a range of values. Total reliability was uniformly high, interval reliability was spuriously high for high-rate responding, proportional reliability was somewhat lower for high-rate responding, and exact-agreement reliability was the lowest of the measures, especially for high-rate responding. In Study 2, we examined the separate effects of response rate per se, bursting, and end-of-interval responding. Response rate and bursting had little effect on reliability scores; however, the distribution of some responses at the end of intervals decreased interval reliability somewhat, proportional reliability noticeably, and exact-agreement reliability markedly.
authors: Kennedy Krieger Institute, USA.
authors: Natalie U
authors: NU
authors: Rolider
authors: Brian A
authors: BA
authors: Iwata
authors: Christopher E
authors: CE
authors: Bullock
mesh_terms: D006801:Humans; D015588:Observer Variation; D011930:Reaction Time; D015203:Reproducibility of Results; D055815:Young Adult
keywords: interobserver agreement; reliability; response distribution; response rate