title: Health system guidance appraisal--concept evaluation and usability testing.
abstract: BACKGROUND
  Health system guidance (HSG) provides recommendations aimed to address health system challenges. However, there is a paucity of methods to direct, appraise, and report HSG. Earlier research identified 30 candidate criteria (concepts) that can be used to evaluate the quality of HSG and guide development and reporting requirements. The objective of this paper was to describe two studies aimed at evaluating the importance of these 30 criteria, design a draft HSG appraisal tool, and test its usability.
  
  
  METHODS
  This study involved a two-step survey process. In step 1, respondents rated the 30 concepts for appropriateness to, relevance to, and priority for health system decisions and HSG. This led to a draft tool. In step 2, respondents reviewed HSG documents, appraised them using the tool, and answered a series of questions. Descriptive analyses were computed.
  
  
  RESULTS
  Fifty participants were invited in step 1, and we had a response rate of 82 %. The mean response rates for each concept within each survey question were universally favorable. There was also an overall agreement about the need for a high-quality tool to systematically direct the development, appraisal, and reporting of HSG. Qualitative feedback and a consensus process by the team led to refinements to some of the concepts and the creation of a beta (draft) version of the HSG tool. In step 2, 35 participants were invited and we had a response rate of 74 %. Exploratory analyses showed that the quality of the HSGs reviewed varied as a function of the HSG item and the specific document assessed. A favorable consensus was reached with participants agreeing that the HSG items were easy to understand and easy to apply. Moreover, the overall agreement was high for the usability of the tool to systematically direct the development (85 %), appraisal (92 %), and reporting (81 %) of HSG. From this process, version 1.0 of the HSG appraisal tool was generated complete with 32 items (and their descriptions) and 4 domains.
  
  
  CONCLUSIONS
  The final tool, named the Appraisal of Guidelines for Research and Evaluation for Health Systems (AGREE-HS) (version 1), defines expectations of HSG and facilitates informed decisions among policymakers on health system delivery, financial, and governance arrangements.
authors.affiliation: McMaster University, Juravinski Hospital Site, G Wing, 2nd Floor, Room 207, 711 Concession Street, Hamilton, Ontario, L8V 1C3, Canada. akoarrde@mcmaster.ca.
authors.forename: Denis E
authors.initials: DE
authors.lastname: Ako-Arrey
authors.affiliation: McMaster University, Juravinski Hospital Site, G Wing, 2nd Floor, Room 207, 711 Concession Street, Hamilton, Ontario, L8V 1C3, Canada. mbrouwer@mcmaster.ca.
authors.forename: Melissa C
authors.initials: MC
authors.lastname: Brouwers
authors.affiliation: McMaster University, MML-417, 1280 Main St. West, Hamilton, ON, L8S 4L6, Canada. lavisj@mcmaster.ca.
authors.forename: John N
authors.initials: JN
authors.lastname: Lavis
authors.affiliation: McMaster University, CRL-218, 1280 Main Street West, Hamilton, ON, L8S 4K1, Canada. giacomin@mcmaster.ca.
authors.forename: Mita K
authors.initials: MK
authors.lastname: Giacomini
mesh_terms: D000328:Adult; D001291:Attitude of Health Personnel; D003695:Delivery of Health Care / Q000592:standards*; D019317:Evidence-Based Medicine / Q000592:standards*; D005260:Female; D017408:Guidelines as Topic; D006282:Health Personnel / Q000523:psychology*; D006293:Health Promotion / Q000592:standards*; D006801:Humans; D008297:Male; D008875:Middle Aged; D011787:Quality of Health Care / Q000592:standards*; D011795:Surveys and Questionnaires