title: An Evolving Ecosystem for Natural Language Processing in Department of Veterans Affairs.
abstract: In an ideal clinical Natural Language Processing (NLP) ecosystem, researchers and developers would be able to collaborate with others, undertake validation of NLP systems, components, and related resources, and disseminate them. We captured requirements and formative evaluation data from the Veterans Affairs (VA) Clinical NLP Ecosystem stakeholders using semi-structured interviews and meeting discussions. We developed a coding rubric to code interviews. We assessed inter-coder reliability using percent agreement and the kappa statistic. We undertook 15 interviews and held two workshop discussions. The main areas of requirements related to; design and functionality, resources, and information. Stakeholders also confirmed the vision of the second generation of the Ecosystem and recommendations included; adding mechanisms to better understand terms, measuring collaboration to demonstrate value, and datasets/tools to navigate spelling errors with consumer language, among others. Stakeholders also recommended capability to: communicate with developers working on the next version of the VA electronic health record (VistA Evolution), provide a mechanism to automatically monitor download of tools and to automatically provide a summary of the downloads to Ecosystem contributors and funders. After three rounds of coding and discussion, we determined the percent agreement of two coders to be 97.2% and the kappa to be 0.7851. The vision of the VA Clinical NLP Ecosystem met stakeholder needs. Interviews and discussion provided key requirements that inform the design of the VA Clinical NLP Ecosystem.
authors.affiliation: IDEAS Center SLC VA Healthcare System, 500 Foothill Drive, Salt Lake City, UT, 84148, USA. jennifer.garvin@va.gov.|GRECC SLC VA Healthcare System, 500 Foothill Drive, Salt Lake City, UT, 84148, USA. jennifer.garvin@va.gov.|Division of Epidemiology, University of Utah School of Medicine, 295 Chipeta Way, Salt Lake City, UT, 84132, USA. jennifer.garvin@va.gov.|Department of Biomedical Informatics, University of Utah School of Medicine, 421 Wakara Way, Ste. 140, Salt Lake City, UT, 84108, USA. jennifer.garvin@va.gov.
authors.forename: Jennifer H
authors.initials: JH
authors.lastname: Garvin
authors.affiliation: IDEAS Center SLC VA Healthcare System, 500 Foothill Drive, Salt Lake City, UT, 84148, USA.|Department of Biomedical Informatics, University of Utah School of Medicine, 421 Wakara Way, Ste. 140, Salt Lake City, UT, 84108, USA.
authors.forename: Megha
authors.initials: M
authors.lastname: Kalsy
authors.affiliation: VA Connecticut Healthcare System, 950 Campbell Avenue, West Haven, CT, USA.|Yale School of Medicine, 333 Cedar St., New Haven, CT, USA.
authors.forename: Cynthia
authors.initials: C
authors.lastname: Brandt
authors.affiliation: James A Haley Veterans Hospital, 13000 Bruce B. Downs Blvd, Tampa, FL, USA.
authors.forename: Stephen L
authors.initials: SL
authors.lastname: Luther
authors.affiliation: IDEAS Center SLC VA Healthcare System, 500 Foothill Drive, Salt Lake City, UT, 84148, USA.|Department of Biomedical Informatics, University of Utah School of Medicine, 421 Wakara Way, Ste. 140, Salt Lake City, UT, 84108, USA.
authors.forename: Guy
authors.initials: G
authors.lastname: Divita
authors.affiliation: IDEAS Center SLC VA Healthcare System, 500 Foothill Drive, Salt Lake City, UT, 84148, USA.
authors.forename: Gregory
authors.initials: G
authors.lastname: Coronado
authors.affiliation: IDEAS Center SLC VA Healthcare System, 500 Foothill Drive, Salt Lake City, UT, 84148, USA.|Department of Biomedical Informatics, University of Utah School of Medicine, 421 Wakara Way, Ste. 140, Salt Lake City, UT, 84108, USA.|Department of Clinical Research and Leadership, George Washington University School of Medicine and Health Sciences, 2100 Pennsylvania Ave, NW, Washington, DC, 20037, USA.
authors.forename: Doug
authors.initials: D
authors.lastname: Redd
authors.affiliation: IDEAS Center SLC VA Healthcare System, 500 Foothill Drive, Salt Lake City, UT, 84148, USA.|Department of Biomedical Informatics, University of Utah School of Medicine, 421 Wakara Way, Ste. 140, Salt Lake City, UT, 84108, USA.
authors.forename: Carrie
authors.initials: C
authors.lastname: Christensen
authors.affiliation: IDEAS Center SLC VA Healthcare System, 500 Foothill Drive, Salt Lake City, UT, 84148, USA.|Department of Biomedical Informatics, University of Utah School of Medicine, 421 Wakara Way, Ste. 140, Salt Lake City, UT, 84108, USA.
authors.forename: Brent
authors.initials: B
authors.lastname: Hill
authors.affiliation: IDEAS Center SLC VA Healthcare System, 500 Foothill Drive, Salt Lake City, UT, 84148, USA.
authors.forename: Natalie
authors.initials: N
authors.lastname: Kelly
authors.affiliation: IDEAS Center SLC VA Healthcare System, 500 Foothill Drive, Salt Lake City, UT, 84148, USA.|Department of Biomedical Informatics, University of Utah School of Medicine, 421 Wakara Way, Ste. 140, Salt Lake City, UT, 84108, USA.|Department of Clinical Research and Leadership, George Washington University School of Medicine and Health Sciences, 2100 Pennsylvania Ave, NW, Washington, DC, 20037, USA.
authors.forename: Qing Zeng
authors.initials: QZ
authors.lastname: Treitler
mesh_terms: D003142:Communication; D003299:Cooperative Behavior; D057286:Electronic Health Records / Q000458:organization & administration* / Q000592:standards; D006801:Humans; D007407:Interviews as Topic; D009323:Natural Language Processing; D015203:Reproducibility of Results; D009626:Terminology as Topic; D014481:United States; D014493:United States Department of Veterans Affairs / Q000458:organization & administration*
keywords: Formative evaluation; Natural language processing