title: Inter-rater reliability for scoring children's dichotic words test responses.
abstract: OBJECTIVE
  To determine whether rater agreement is randomly distributed or varies importantly with test-taker characteristics, test words, or rater experience with the dichotic words test (DWT).
  
  
  METHODS
  DWT was administered to 34 children in 1st-4th grade and responses scored by two raters. The proportion of rater agreement was calculated for each child and for each word. Correlates of inter-rater agreement were explored.
  
  
  METHODS
  Two raters judged 6686 total responses from 34 children.
  
  
  RESULTS
  Overall agreement between the two raters was 0.97. Test-taker scores ranged from 35%-91% (mean = 81%). Agreement was associated with score but not with test-taker age or sex. Test words spanned the full range of difficulty (pass proportion 0.06-1.00). Rater agreement was not randomly distributed among the words. Inter-rater agreement for test words ranged from 0.82-1.00 and was associated with pass proportion (Spearman's ρ = 0.28; p < 0.0001). However, there were words at all pass proportions with perfect or near-perfect agreement. Rater agreement improved from 0.94 on the first day of data collection to 0.98 on the fifth day (p = 0.026).
  
  
  CONCLUSIONS
  Inter-rater reliability should be considered along with test item difficulty when developing speech audiometry materials, scoring protocols, and rater training.
authors.affiliation: a Clinical and Translational Science, University of Vermont , Burlington , USA.
authors.forename: Kairn Stetler
authors.initials: KS
authors.lastname: Kelley
authors.affiliation: a Clinical and Translational Science, University of Vermont , Burlington , USA.
authors.forename: Benjamin
authors.initials: B
authors.lastname: Littenberg
mesh_terms: D000161:Acoustic Stimulation; D000367:Age Factors; D001302:Audiometry, Speech; D001309:Auditory Threshold; D002648:Child; D002652:Child Behavior; D004007:Dichotic Listening Tests; D019988:Children with Disabilities / Q000523:psychology*; D005260:Female; D034381:Hearing Loss / Q000175:diagnosis* / Q000503:physiopathology / Q000523:psychology; D006801:Humans; D008297:Male; D015588:Observer Variation; D019986:Persons with Hearing Disabilities / Q000523:psychology*; D011237:Predictive Value of Tests; D015203:Reproducibility of Results; D013067:Speech Perception
keywords: Dichotic listening tests; reproducibility of results; speech audiometry