How large must a treatment effect be before it matters to practitioners? An estimation method and demonstration.
OBJECTIVE
Treatment research is sometimes criticised as lacking in clinical relevance, and one potential source of this friction is a disconnection between statistical significance and what clinicians regard to be a meaningful difference in outcomes. This report demonstrates a novel methodology for estimating what substance abuse practitioners regard to be clinically important differences.


METHODS
To illustrate the estimation method, we surveyed 50 substance abuse treatment providers participating in the National Institute on Drug Abuse (NIDA) Clinical Trials Network. Practitioners identified thresholds for clinically meaningful differences on nine common outcome variables, indicated the size of effect that would justify their learning a new treatment method and estimated current outcomes from their services.


RESULTS
Clinicians judged a difference between two treatments to be meaningful if outcomes were improved by about 10 - 12 points on the percentage of patients totally abstaining, arrested for driving while intoxicated, employed or having abnormal liver enzymes. A 5 percentage-point reduction in patient mortality was regarded as clinically significant. On continuous outcome measures (such as percentage of days abstinent or drinks per drinking day), practitioners judged an outcome to be significant when it doubled or halved the base rate. When a new treatment meets such criteria, practitioners were interested in learning it.


CONCLUSIONS
Effects that are statistically significant in clinical trials may be unimpressive to practitioners. Clinicians' judgements of meaningful differences can inform the powering of clinical trials.
Miller
William R
WR
Center on Alcoholism, Substance Abuse and Addictions (CASAA), The University of New Mexico, Albuquerque, New Mexico, USA. wrmiller@unm.edu
Manuel
Jennifer Knapp
JK
D003627:Data Interpretation, Statistical; D019317:Evidence-Based Medicine / Q000379:methods* / Q000706:statistics & numerical data; D006282:Health Personnel / Q000523:psychology*; D006801:Humans; D017063:Outcome Assessment, Health Care / Q000379:methods* / Q000706:statistics & numerical data; D012016:Reference Values; D015203:Reproducibility of Results