title: Inter-observer reliability assessments in time motion studies: the foundation for meaningful clinical workflow analysis.
abstract: Understanding clinical workflow is critical for researchers and healthcare decision makers. Current workflow studies tend to oversimplify and underrepresent the complexity of clinical workflow. Continuous observation time motion studies (TMS) could enhance clinical workflow studies by providing rich quantitative data required for in-depth workflow analyses. However, methodological inconsistencies have been reported in continuous observation TMS, potentially reducing the validity of TMS' data and limiting their contribution to the general state of knowledge. We believe that a cornerstone in standardizing TMS is to ensure the reliability of the human observers. In this manuscript we review the approaches for inter-observer reliability assessment (IORA) in a representative sample of TMS focusing on clinical workflow. We found that IORA is an uncommon practice, inconsistently reported, and often uses methods that provide partial and overestimated measures of agreement. Since a comprehensive approach to IORA is yet to be proposed and validated, we provide initial recommendations for IORA reporting in continuous observation TMS.
authors.affiliation: Department of Biomedical Informatics, The Ohio State University, Columbus, OH, USA.
authors.forename: Marcelo A
authors.initials: MA
authors.lastname: Lopetegui
authors.affiliation: Division of Biostatistics, College of Public Health, The Ohio State University, Columbus, OH, USA.
authors.forename: Shasha
authors.initials: S
authors.lastname: Bai
authors.affiliation: Department of Biomedical Informatics, The Ohio State University, Columbus, OH, USA.
authors.forename: Po-Yin
authors.initials: PY
authors.lastname: Yen
authors.affiliation: Department of Biomedical Informatics, The Ohio State University, Columbus, OH, USA.
authors.forename: Albert
authors.initials: A
authors.lastname: Lai
authors.affiliation: Department of Biomedical Informatics, The Ohio State University, Columbus, OH, USA.
authors.forename: Peter
authors.initials: P
authors.lastname: Embi
authors.affiliation: Department of Biomedical Informatics, The Ohio State University, Columbus, OH, USA.
authors.forename: Philip R O
authors.initials: PR
authors.lastname: Payne
mesh_terms: D006739:Hospital Administration; D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results; D013996:Time and Motion Studies; D057188:Workflow