Reliability issues and solutions for coding social communication performance in classroom settings.
OBJECTIVE
To explore the utility of time-interval analysis for documenting the reliability of coding social communication performance of children in classroom settings. Of particular interest was finding a method for determining whether independent observers could reliably judge both occurrence and duration of ongoing behavioral dimensions for describing social communication performance.


METHODS
Four coders participated in this study. They observed and independently coded 6 social communication behavioral dimensions using handheld computers. The dimensions were mutually exclusive and accounted for all verbal and nonverbal productions during a specified time frame. The technology allowed for coding frequency and duration for each entered code. Data were collected from 20 different 2-min video segments of children in kindergarten through 3rd-grade classrooms. Data were analyzed for interobserver and intraobserver agreements using time-interval sorting and Cohen's kappa. Further, interval size and total observation length were manipulated to determine their influence on reliability.


RESULTS
The data revealed interval sorting and kappa to be a suitable method for examining reliability of occurrence and duration of ongoing social communication behavioral dimensions. Nearly all comparisons yielded medium to large kappa values; interval size and length of observation minimally affected results. Implications The analysis procedure described in this research solves a challenge in reliability: comparing coding by independent observers of both occurrence and duration of behaviors. Results indicate the utility of a new coding taxonomy and technology for application in online observations of social communication in a classroom setting.
Olswang
Lesley B
LB
Department of Speech and Hearing Sciences, University of Washington, 1417 Northeast 42nd Street, Seattle, WA 98105-6246l, USA. lolswang@u.washington.edu
Svensson
Liselotte
L
Coggins
Truman E
TE
Beilinson
Jill S
JS
Donaldson
Amy L
AL
D000328:Adult; D002648:Child; D003142:Communication; D006801:Humans; D007398:Interpersonal Relations; D008875:Middle Aged; D015588:Observer Variation; D010379:Peer Group; D015203:Reproducibility of Results; D012919:Social Behavior; D014705:Verbal Behavior