title: Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory.
abstract: BACKGROUND
  The National Institutes of Health (NIH) Health Care Systems Research Collaboratory (NIH Collaboratory) seeks to produce generalizable knowledge about the conduct of pragmatic research in health systems. This analysis applied the PRECIS-2 pragmatic trial criteria to five NIH Collaboratory pragmatic trials to better understand 1) the pragmatic aspects of the design and implementation of treatments delivered in real world settings and 2) the usability of the PRECIS-2 criteria for assessing pragmatic features across studies and across time.
  
  
  METHODS
  Using the PRECIS-2 criteria, five pragmatic trials were each rated by eight raters. For each trial, we reviewed the original grant application and a required progress report written at the end of a 1-year planning period that included changes to the protocol or implementation approach. We calculated median scores and interrater reliability for each PRECIS domain and for the overall trial at both time points, as well as the differences in scores between the two time points. We also reviewed the rater comments associated with the scores.
  
  
  RESULTS
  All five trials were rated to be more pragmatic than explanatory, with comments indicating that raters generally perceived them to closely mirror routine clinical care across multiple domains. The PRECIS-2 domains for which the trials were, on average, rated as most pragmatic on the 1 to 5 scale at the conclusion of the planning period included primary analysis (mean = 4.7 (range = 4.5 to 4.9)), recruitment (4.3 (3.6 to 4.8)), eligibility (4.1 (3.4 to 4.8)), setting (4.1 (4.0 to 4.4)), follow-up (4.1 (3.4 to 4.9)), and primary outcome (4.1 (3.5 to 4.9)). On average, the less pragmatic domains were organization (3.3 (2.6 to 4.4)), flexibility of intervention delivery (3.5 (2.1-4.5)), and flexibility of intervention adherence (3.8 (2.8-4.5)). Interrater agreement was modest but statistically significant for four trials (Gwet's AC1 statistic range 0.23 to 0.40) and the intraclass correlation coefficient ranged from 0.05 to 0.31. Rating challenges included assigning a single score for domains that may relate to both patients and care settings (that is, eligibility or recruitment) and determining to what extent aspects of complex research interventions differ from usual care.
  
  
  CONCLUSIONS
  These five trials in diverse healthcare settings were rated as highly pragmatic using the PRECIS-2 criteria. Applying the tool generated insightful discussion about real-world design decisions but also highlighted challenges using the tool. PRECIS-2 raters would benefit from additional guidance about how to rate the interwoven patient and practice-level considerations that arise in pragmatic trials.
  
  
  BACKGROUND
  Clinicaltrials.gov trial registrations: NCT02019225 , NCT01742065 , NCT02015455 , NCT02113592 , NCT02063867 .
authors.affiliation: Group Health Research Institute, Seattle, USA. karinjohnso@gmail.com.
authors.forename: Karin E
authors.initials: KE
authors.lastname: Johnson
authors.affiliation: National Cancer Institute, Bethesda, MD, USA. netagil@mail.nih.gov.
authors.forename: Gila
authors.initials: G
authors.lastname: Neta
authors.affiliation: University of Pennsylvania Perelman School of Medicine, Philadelphia, USA. ldember@upenn.edu.
authors.forename: Laura M
authors.initials: LM
authors.lastname: Dember
authors.affiliation: Kaiser Permanente Center for Health Research, Portland, OR, USA. gloria.d.coronado@kpchr.org.
authors.forename: Gloria D
authors.initials: GD
authors.lastname: Coronado
authors.affiliation: National Cancer Institute, Bethesda, MD, USA. jerry.suls@nih.gov.
authors.forename: Jerry
authors.initials: J
authors.lastname: Suls
authors.affiliation: National Cancer Institute, Bethesda, MD, USA. dchamber@mail.nih.gov.
authors.forename: David A
authors.initials: DA
authors.lastname: Chambers
authors.affiliation: University of Washington, Seattle, WA, USA. srundell@u.washington.edu.
authors.forename: Sean
authors.initials: S
authors.lastname: Rundell
authors.affiliation: Kaiser Permanente Center for Health Research, Portland, OR, USA. David.H.Smith@kpchr.org.
authors.forename: David H
authors.initials: DH
authors.lastname: Smith
authors.affiliation: National Cancer Institute, Bethesda, MD, USA. liub2@mail.nih.gov.
authors.forename: Benmei
authors.initials: B
authors.lastname: Liu
authors.affiliation: National Cancer Institute, Bethesda, MD, USA. taplins@mail.nih.gov.
authors.forename: Stephen
authors.initials: S
authors.lastname: Taplin
authors.affiliation: National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, MD, USA. stoneyc@nhlbi.nih.gov.
authors.forename: Catherine M
authors.initials: CM
authors.lastname: Stoney
authors.affiliation: National Cancer Institute, Bethesda, MD, USA. farrellm@mail.nih.gov.
authors.forename: Margaret M
authors.initials: MM
authors.lastname: Farrell
authors.affiliation: Department of Family Medicine, University of Colorado School of Medicine, Denver, USA. Russell.Glasgow@ucdenver.edu.
authors.forename: Russell E
authors.initials: RE
authors.lastname: Glasgow
mesh_terms: D002986:Clinical Trials as Topic; D003299:Cooperative Behavior; D003695:Delivery of Health Care; D006801:Humans; D009316:National Institutes of Health (U.S.); D015203:Reproducibility of Results; D012107:Research Design; D014481:United States