A scale for distinguishing efficacy from effectiveness was adapted and applied to stroke rehabilitation studies.
OBJECTIVE
To apply a tool that purports to differentiate between efficacy and effectiveness studies to stroke rehabilitation trials and to evaluate its applicability and reliability.


METHODS
Three raters developed item operational definitions before independently applying the seven-item scale to 151 randomized controlled trials (RCT), published during or after 1997, that evaluated either a pharmacologic (P, n=78) or a nonpharmacologic (NP, n=73) intervention. Inter-rater reliability was assessed for both individual items and total scores, separately for P and NP trials.


RESULTS
Item inter-rater reliability (multiple-rater kappa) ranged from 0.00 (95% CI [confidence interval]: -0.13, 0.13) to 0.85 (95% CI: 0.73, 0.98) and from 0.21 (95% CI: 0.08, 0.34) to 0.79 (95% CI: 0.66, 0.92) for P and NP RCTs, respectively. For the total score (dichotomized), kappa values were 0.43 (95% CI: 0.31, 0.56) and 0.51 (95% CI: 0.37, 0.64) for P and NP trials, respectively.


CONCLUSIONS
The tool provides a solid foundation on which to base further discussion of the differential criteria of efficacy-effectiveness trial design. Scale items should be properly operationalized depending on the research question of interest and evaluated for reliability before the scale is used for definitively judging a given study's design or the external validity of its results.
Zettler
Laura L
LL
Department of Epidemiology and Biostatistics, Schulich School of Medicine and Dentistry, The University of Western Ontario, London, Ontario, Canada. lauraz@chatham-kent.ca
Speechley
Mark R
MR
Foley
Norine C
NC
Salter
Katherine L
KL
Teasell
Robert W
RW
D019317:Evidence-Based Medicine / Q000379:methods; D006801:Humans; D015588:Observer Variation; D016032:Randomized Controlled Trials as Topic / Q000592:standards*; D015203:Reproducibility of Results; D012107:Research Design; D000071939:Stroke Rehabilitation; D016896:Treatment Outcome