title: Very large treatment effects in randomised trials as an empirical marker to indicate whether subsequent trials are necessary: meta-epidemiological assessment.
abstract: OBJECTIVE
  To examine whether a very large effect (VLE; defined as a relative risk of ≤0.2 or ≥5) in a randomised trial could be an empirical marker that subsequent trials are unnecessary.
  
  
  METHODS
  Meta-epidemiological assessment of existing published data on randomised trials.
  
  
  METHODS
  Cochrane Database of Systematic Reviews (2010, issue 7) with data on subsequent large trials updated to 2015, issue 12.
  
  
  METHODS
  All binary outcome forest plots were selected, which contained an index randomised trial with a VLE that was nominally statistically significant (P<0.05), included a subsequent large randomised trial (≥200 events and ≥200 non-events) for validation of the effect, assessed a primary outcome of the review, and was not a subgroup or sensitivity analysis.
  
  
  RESULTS
  Of 3082 reviews yielding 85 002 forest plots, only 44 (0.05%) satisfied the inclusion criteria. Index trials were generally small, with a median sample of 99 (median 14 events). Few index trials were rated at low risk of bias (9 of 44; 20%). The relative risk was closer to the null in the subsequent large trials in 43 of 44 cases. Subsequent large trial data failed to find a statistically significant (P<0.05) effect in the same direction in 19 cases (43%, 95% confidence interval 29% to 58%). Even when the subsequent large trials did find a significant effect in the same direction, the additional primary outcomes in most of these trials would have to be considered before deciding in favour of using the intervention. Subsequent large trial data found a statistically significant effect in the same direction in 19 of 21 cases when the index trial also had a value of P<0.001.
  
  
  CONCLUSIONS
  The frequency of VLEs followed by a large trial is vanishingly small, and where they occur they do not appear to be a reliable marker for a benefit that is reproducible and directly actionable. An empirical rule using a VLE in a randomised controlled trial as a marker that further trials are unnecessary would be neither practical nor useful. Caution should be taken when interpreting small studies with very large treatment effects.
authors: Division of Anaesthetics, Pain Medicine and Intensive Care, Imperial College London, UK.
authors: Myura
authors: M
authors: Nagendran
authors: Health Technology Assessment Unit, Institute of Education and Sciences, Hospital Alemão Oswaldo Cruz, Sao Paulo, Brazil.
authors: Tiago V
authors: TV
authors: Pereira
authors: Gonville and Caius College, University of Cambridge, UK.
authors: Grace
authors: G
authors: Kiew
authors: Centre for Statistics in Medicine, Oxford, UK.
authors: Douglas G
authors: DG
authors: Altman
authors: Department of Epidemiology and Public Health, University College London, UK.
authors: Mahiben
authors: M
authors: Maruthappu
authors: Departments of Medicine, of Health Research and Policy, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University, USA.
authors: John P A
authors: JP
authors: Ioannidis
authors: Nuffield Department of Surgical Science, John Radcliffe Hospital, University of Oxford, Oxford OX3 9DU, UK peter.mcculloch@nds.ox.ac.uk.
authors: Peter
authors: P
authors: McCulloch
mesh_terms: D003627:Data Interpretation, Statistical; D006801:Humans; D016032:Randomized Controlled Trials as Topic / Q000706:statistics & numerical data; D015203:Reproducibility of Results; D016896:Treatment Outcome