A framework to assess the value of application of formal criteria to check clinical relevance in RCTs as part of a benefit assessment strategy.
Recently, the topic of assessing clinical relevance on top of statistical significance in the analysis of randomized control trials (RCTs) has got increasing attention, in particular as part of benefit assessments. Several formal criteria to serve this purpose have been published. In this paper, we present a framework to assess the value of the application of such criteria. We propose to quantify the need for the assessment of clinical relevance by the actual risk of having accepted a benefit for a treatment with an irrelevant effect in a successful RCT. We then study how this risk can be controlled by two popular criteria based on comparing the effect estimate or the lower bound of the confidence interval with a given threshold. We further propose to quantify the impact of using formal criteria by considering the expected costs when specifying error-specific costs for each of the three possible types of errors: A benefit may be accepted for a treatment, which is actually inferior, or which is not inferior, but only implies an irrelevant improvement, or a benefit may be rejected for a treatment implying a relevant improvement. This way we can demonstrate that the impact depends on parameters which are typically not explicitly defined in the frame of benefit assessments. Depending on the values of these parameters, formal checks of clinical relevance may imply better decisions on average, but they may also imply more harm than good on average.
Clinical Epidemiology, Center of Medical Biometry and Medical Informatics, University Medical Center Freiburg, Stefan Meier Str. 26, D-79104, Freiburg, Germany.
Werner
W
Vach
Division of Infectious Diseases, Tübingen University Hospital, Otfried-Müller-Strae 12, D-72076, Tübingen, Germany.
Beryl Primrose
BP
Gladstone
D001699:Biometry / Q000379:methods*; D005069:Evaluation Studies as Topic; D006801:Humans; D011336:Probability; D016032:Randomized Controlled Trials as Topic; D018570:Risk Assessment; D018401:Sample Size
Clinical relevance; Power; RCT; Sample size; Shifted null hypothesis