title: The American Academy of Sleep Medicine Inter-scorer Reliability program: respiratory events.
abstract: OBJECTIVE
  The American Academy of Sleep Medicine (AASM) Inter-scorer Reliability program provides a unique opportunity to compare a large number of scorers with varied levels of experience to determine agreement in the scoring of respiratory events. The objective of this paper is to examine areas of disagreement to inform future revisions of the AASM Manual for the Scoring of Sleep and Associated Events.
  
  
  METHODS
  The sample included 15 monthly records, 200 epochs each. The number of scorers increased steadily during the period of data collection, reaching more than 3,600 scorers by the final record. Scorers were asked to identify whether an obstructive, mixed, or central apnea; a hypopnea; or no event was seen in each of the 200 epochs. The "correct" respiratory event score was defined as the score endorsed by the most scorers. Percentage agreement with the majority score was determined for each epoch and the mean agreement determined.
  
  
  RESULTS
  The overall agreement for scoring of respiratory events was 93.9% (κ = 0.92). There was very high agreement on epochs without respiratory events (97.4%), and the majority score for most of the epochs (87.8%) was no event. For the 364 epochs scored as having a respiratory event, overall agreement that some type of respiratory event occurred was 88.4% (κ = 0.77). The agreement for epochs scored as obstructive apnea by the majority was 77.1% (κ = 0.71), and the most common disagreement was hypopnea rather than obstructive apnea (14.4%). The agreement for hypopnea was 65.4% (κ = 0.57), with 16.4% scoring no event and 14.8% scoring obstructive apnea. The agreement for central apnea was 52.4% (κ = 0.41). A single epoch was scored as a mixed apnea by a plurality of scorers.
  
  
  CONCLUSIONS
  The study demonstrated excellent agreement among a large sample of scorers for epochs with no respiratory events. Agreement for some type of event was good, but disagreements in scoring of apnea vs. hypopnea and type of apnea were common. A limitation of the analysis is that most of the records had normal breathing. A review of controversial events yielded no consistent bias that might be resolved by a change of scoring rules.
authors.affiliation: California State University, Long Beach, CA.
authors.forename: Richard S
authors.initials: RS
authors.lastname: Rosenberg
authors.affiliation: American Academy of Sleep Medicine, Darien, IL.
authors.forename: Steven
authors.initials: S
authors.lastname: Van Hout
mesh_terms: D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results; D012143:Respiratory Physiological Phenomena; D012891:Sleep Apnea Syndromes / Q000175:diagnosis* / Q000503:physiopathology; D056508:Sleep Medicine Specialty / Q000592:standards*; D012955:Societies, Medical / Q000592:standards; D014481:United States
keywords: Scoring; apnea; hypopnea; reliability; respiratory events
references.citation: Chest. 2005 Jan;127(1):80-8
references.pmid: 15653966
references.citation: Sleep. 2013 Apr 01;36(4):591-6
references.pmid: 23565005
references.citation: J Clin Sleep Med. 2012 Oct 15;8(5):597-619
references.pmid: 23066376
references.citation: J Clin Sleep Med. 2013 Jan 15;9(1):81-7
references.pmid: 23319910
references.citation: Sleep. 2004 Nov 1;27(7):1394-403
references.pmid: 15586793
references.citation: J Clin Sleep Med. 2007 Mar 15;3(2):169-200
references.pmid: 17557426
references.citation: Sleep. 1998 Nov 1;21(7):749-57
references.pmid: 11286351