title: Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review.
abstract: BACKGROUND
  Systematic literature reviews identify, select, appraise, and synthesize relevant literature on a particular topic. Typically, these reviews examine primary studies based on similar methods, e.g., experimental trials. In contrast, interest in a new form of review, known as mixed studies review (MSR), which includes qualitative, quantitative, and mixed methods studies, is growing. In MSRs, reviewers appraise studies that use different methods allowing them to obtain in-depth answers to complex research questions. However, appraising the quality of studies with different methods remains challenging. To facilitate systematic MSRs, a pilot Mixed Methods Appraisal Tool (MMAT) has been developed at McGill University (a checklist and a tutorial), which can be used to concurrently appraise the methodological quality of qualitative, quantitative, and mixed methods studies.
  
  
  OBJECTIVE
  The purpose of the present study is to test the reliability and efficiency of a pilot version of the MMAT.
  
  
  METHODS
  The Center for Participatory Research at McGill conducted a systematic MSR on the benefits of Participatory Research (PR). Thirty-two PR evaluation studies were appraised by two independent reviewers using the pilot MMAT. Among these, 11 (34%) involved nurses as researchers or research partners. Appraisal time was measured to assess efficiency. Inter-rater reliability was assessed by calculating a kappa statistic based on dichotomized responses for each criterion. An appraisal score was determined for each study, which allowed the calculation of an overall intra-class correlation.
  
  
  RESULTS
  On average, it took 14 min to appraise a study (excluding the initial reading of articles). Agreement between reviewers was moderate to perfect with regards to MMAT criteria, and substantial with respect to the overall quality score of appraised studies.
  
  
  CONCLUSIONS
  The MMAT is unique, thus the reliability of the pilot MMAT is promising, and encourages further development.
authors.affiliation: Department of Family Medicine, McGill University, 517 Pine Avenue West, Montreal, QC, Canada H2W 1S4. pierre.pluye@mcgill.ca
authors.forename: Romina
authors.initials: R
authors.lastname: Pace
authors.forename: Pierre
authors.initials: P
authors.lastname: Pluye
authors.forename: Gillian
authors.initials: G
authors.lastname: Bartlett
authors.forename: Ann C
authors.initials: AC
authors.lastname: Macaulay
authors.forename: Jon
authors.initials: J
authors.lastname: Salsberg
authors.forename: Justin
authors.initials: J
authors.lastname: Jagosh
authors.forename: Robbyn
authors.initials: R
authors.lastname: Seller
mesh_terms: D035843:Biomedical Research; D017598:Efficiency, Organizational; D010865:Pilot Projects; D015203:Reproducibility of Results; D000078202:Systematic Reviews as Topic