The undue influence of significant p-values on the perceived importance of study results.
BACKGROUND
Statistically significant differences between treatments (i.e., results typically associated with p < 0.05) may not always correspond to important differences upon which to base orthopedic practice. If the hypothesis that p < 0.05 unduly influences the perception of importance of study results were true, we would expect that presenting such a p-value would lead to 1) greater agreement among clinicians about the importance of a study result, and 2) greater perceived importance of a study result, when compared with presenting the same results omitting the p-value.


METHODS
The participants were 3 orthopedics residents, 5 fellows, and 4 attending orthopedic surgeons at a university hospital. We constructed a 40-item questionnaire with the comparison groups, primary outcome of interest, and the results from each of 40 studies. These studies represent a variety of interventions across orthopedic surgery assessed in 2-group comparative intervention studies (randomized trials, observational studies) and were published between 2000 and 2002 in the Journal of Bone and Joint Surgery--American Volume. For each question, respondents were asked to rate the importance of the study results. Participants answered the questionnaire first without p-values and then, 8 weeks later, with p-values (and a random sample of items without p-values). An intra-class correlation quantified agreement between clinicians when answering items with and without p-values. The difference in mean importance scores between the two presentations was also estimated.


RESULTS
Of 40 eligible clinical comparative studies, 30 reported p < 0.05 for their primary comparison. Without presenting p-values, overall agreement regarding clinical significance among reviewers was fair (ICC = 0.35, 95% CI: 0.25-0.49). In the 30 studies with p < 0.05, mean importance scores (1 = low; 3 = high) were greater when p-values were presented (difference 0.6, CI 0.1-1.1). 10 of 12 reviewers perceived results to be more important when presented with significant p-values.


CONCLUSIONS
When significant, p-values unduly influence the perception of clinicians regarding the importance of study results.
Bhandari
Mohit
M
Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada. bhandari@sympatico.ca
Montori
Victor M
VM
Schemitsch
Emil H
EH
D035843:Biomedical Research / Q000592:standards / Q000706:statistics & numerical data*; D002986:Clinical Trials as Topic / Q000592:standards*; D003627:Data Interpretation, Statistical; D019317:Evidence-Based Medicine; D006801:Humans; D009985:Orthopedics / Q000592:standards*; D011336:Probability; D011795:Surveys and Questionnaires