title: Semantic Krippendorff's α for measuring inter-rater agreement in SNOMED CT coding studies.
abstract: Semantic interoperability requires consistency in use of terminologies such as SNOMED CT. Inter-rater agreement measurement can be used to quantify this consistency among terminology users. Increasingly, studies of SNOMED CT include inter-rater agreement measures. However, published studies do not consider distance between concepts when calculating the inter-rater agreement measures. In this paper we propose a semantic inter-rater agreement measure for use with SNOMED CT encoded data. A semantic Krippendorff's α measure is implemented using a path-length based difference function. The measure is tested using three different datasets. Results show that the proposed semantic measure is sensitive to seriousness of coding differences whereas a nominal measure is not. The proposed measure reflects the intuition that distance matters when comparing uses of SNOMED CT.
authors.affiliation: Department of Biomedical Engineering, Linköping University.
authors.forename: Daniel
authors.initials: D
authors.lastname: Karlsson
authors.affiliation: Department of Health Science and Technology, Aalborg University.
authors.forename: Kirstine Rosenbeck
authors.initials: KR
authors.lastname: Gøeg
authors.affiliation: Department of Biomedical Engineering, Linköping University.
authors.forename: Håkan
authors.initials: H
authors.lastname: Örman
authors.affiliation: Department of Health Science and Technology, Aalborg University.
authors.forename: Anne Randorff
authors.initials: AR
authors.lastname: Højen
mesh_terms: D000465:Algorithms; D057286:Electronic Health Records / Q000145:classification* / Q000706:statistics & numerical data*; D009323:Natural Language Processing; D015588:Observer Variation; D010363:Pattern Recognition, Automated / Q000379:methods*; D012660:Semantics; D039061:Systematized Nomenclature of Medicine