title: Reliability analysis for a proposed critical appraisal tool demonstrated value for diverse research designs.
abstract: OBJECTIVE
  To examine the reliability of scores obtained from a proposed critical appraisal tool (CAT).
  
  
  METHODS
  Based on a random sample of 24 health-related research papers, the scores from the proposed CAT were examined using intraclass correlation coefficients (ICCs), generalizability theory, and participants' feedback.
  
  
  RESULTS
  The ICC for all research papers was 0.83 (consistency) and 0.74 (absolute agreement) for four participants. For individual research designs, the highest ICC (consistency) was for qualitative research (0.91) and the lowest was for descriptive, exploratory and observational research (0.64). The G study showed a moderate research design effect (32%) for scores averaged across all papers. The research design effect was mainly in the Sampling, Results, and Discussion categories (44%, 36%, and 34%, respectively). The scores for research designs showed a majority paper effect for each (53-70%), with small to moderate rater or paper×rater interaction effects (0-27%).
  
  
  CONCLUSIONS
  Possible reasons for the research design effect were that the participants were unfamiliar with some of the research designs and that papers were not matched to participants' expertise. Even so, the proposed CAT showed great promise as a tool that can be used across a wide range of research designs.
authors.affiliation: Discipline of Physiotherapy, James Cook University, Townsville, QLD 4810, Australia. michael.crowe@jcu.edu.au
authors.forename: Michael
authors.initials: M
authors.lastname: Crowe
authors.forename: Lorraine
authors.initials: L
authors.lastname: Sheppard
authors.forename: Alistair
authors.initials: A
authors.lastname: Campbell
mesh_terms: D001315:Australia; D015340:Epidemiologic Research Design; D019317:Evidence-Based Medicine / Q000592:standards*; D030141:Feedback, Psychological; D006801:Humans; D015203:Reproducibility of Results; D012494:Sampling Studies; D013223:Statistics as Topic