title: Classifying publications from the clinical and translational science award program along the translational research spectrum: a machine learning approach.
abstract: Translational research is a key area of focus of the National Institutes of Health (NIH), as demonstrated by the substantial investment in the Clinical and Translational Science Award (CTSA) program. The goal of the CTSA program is to accelerate the translation of discoveries from the bench to the bedside and into communities. Different classification systems have been used to capture the spectrum of basic to clinical to population health research, with substantial differences in the number of categories and their definitions. Evaluation of the effectiveness of the CTSA program and of translational research in general is hampered by the lack of rigor in these definitions and their application. This study adds rigor to the classification process by creating a checklist to evaluate publications across the translational spectrum and operationalizes these classifications by building machine learning-based text classifiers to categorize these publications.
  
  
  
  Based on collaboratively developed definitions, we created a detailed checklist for categories along the translational spectrum from T0 to T4. We applied the checklist to CTSA-linked publications to construct a set of coded publications for use in training machine learning-based text classifiers to classify publications within these categories. The training sets combined T1/T2 and T3/T4 categories due to low frequency of these publication types compared to the frequency of T0 publications. We then compared classifier performance across different algorithms and feature sets and applied the classifiers to all publications in PubMed indexed to CTSA grants. To validate the algorithm, we manually classified the articles with the top 100 scores from each classifier.
  
  
  
  The definitions and checklist facilitated classification and resulted in good inter-rater reliability for coding publications for the training set. Very good performance was achieved for the classifiers as represented by the area under the receiver operating curves (AUC), with an AUC of 0.94 for the T0 classifier, 0.84 for T1/T2, and 0.92 for T3/T4.
  
  
  
  The combination of definitions agreed upon by five CTSA hubs, a checklist that facilitates more uniform definition interpretation, and algorithms that perform well in classifying publications along the translational spectrum provide a basis for establishing and applying uniform definitions of translational research categories. The classification algorithms allow publication analyses that would not be feasible with manual classification, such as assessing the distribution and trends of publications across the CTSA network and comparing the categories of publications and their citations to assess knowledge transfer across the translational research spectrum.
authors: Health Sciences Library, NYU School of Medicine, New York, USA. alisa.surkis@med.nyu.edu.
authors: Alisa
authors: 0000-0002-9777-2693
authors: A
authors: Surkis
authors: Institute for Clinical and Translational Research, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, USA.
authors: Janice A
authors: JA
authors: Hogle
authors: School of Medicine, Virginia Commonwealth University, Richmond, USA.
authors: Deborah
authors: D
authors: DiazGranados
authors: Indiana Clinical and Translational Sciences Institute, Indiana University School of Medicine, Indianapolis, USA.
authors: Joe D
authors: JD
authors: Hunt
authors: School of Medicine, Virginia Commonwealth University, Richmond, USA.
authors: Paul E
authors: PE
authors: Mazmanian
authors: Clinical and Translational Science Institute, Medical College of Wisconsin, Milwaukee, USA.
authors: Emily
authors: E
authors: Connors
authors: Wisconsin Partnership Program, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, USA.
authors: Kate
authors: K
authors: Westaby
authors: Ruth Lilly Medical Library, Indiana University School of Medicine, Indianapolis, USA.
authors: Elizabeth C
authors: EC
authors: Whipple
authors: Ebling Library for the Health Sciences, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, USA.
authors: Trisha
authors: T
authors: Adamus
authors: Population Health Sciences, School of Medicine and Public Health, University of Wisconsin-Madison, Madison, USA.
authors: Meridith
authors: M
authors: Mueller
authors: Department of Population Health, NYU School of Medicine, New York, USA.
authors: Yindalon
authors: Y
authors: Aphinyanaphongs
mesh_terms: D000465:Algorithms; D019540:Area Under Curve; D004282:Documentation; D000069550:Machine Learning; D011642:Publications / Q000145:classification*; D057170:Translational Research, Biomedical
keywords: Knowledge translation; Machine learning; Text classification; Translational research