title: Average effect estimates remain similar as evidence evolves from single trials to high-quality bodies of evidence: a meta-epidemiologic study.
abstract: OBJECTIVE
  The objective of our study was to use a diverse sample of medical interventions to assess empirically whether first trials rendered substantially different treatment effect estimates than reliable, high-quality bodies of evidence.
  
  
  METHODS
  We used a meta-epidemiologic study design using 100 randomly selected bodies of evidence from Cochrane reports that had been graded as high quality of evidence. To determine the concordance of effect estimates between first and subsequent trials, we applied both quantitative and qualitative approaches. For quantitative assessment, we used Lin's concordance correlation and calculated z-scores; to determine the magnitude of differences of treatment effects, we calculated standardized mean differences (SMDs) and ratios of relative risks. We determined qualitative concordance based on a two-tiered approach incorporating changes in statistical significance and magnitude of effect.
  
  
  RESULTS
  First trials both overestimated and underestimated the true treatment effects in no discernible pattern. Nevertheless, depending on the definition of concordance, effect estimates of first trials were concordant with pooled subsequent studies in at least 33% but up to 50% of comparisons. The pooled magnitude of change as bodies of evidence advanced from single trials to high-quality bodies of evidence was 0.16 SMD [95% confidence interval (CI): 0.12, 0.21]. In 80% of comparisons, the difference in effect estimates was smaller than 0.5 SMDs. In first trials with large treatment effects (>0.5 SMD), however, estimates of effect substantially changed as new evidence accrued (mean change 0.68 SMD; 95% CI: 0.50, 0.86).
  
  
  CONCLUSIONS
  Results of first trials often change, but the magnitude of change, on average, is small. Exceptions are first trials that present large treatment effects, which often dissipate as new evidence accrues.
authors.affiliation: Department for Evidence-based Medicine and Clinical Epidemiology, Danube University, Dr. Karl Dorrek Strasse 30, 3500 Krems, Austria; RTI-International, 3040 East Cornwallis Road, Research Triangle Park, NC 27709-2194, USA. Electronic address: gerald.gartlehner@donau-uni.ac.at.
authors.forename: Gerald
authors.initials: G
authors.lastname: Gartlehner
authors.affiliation: Victor Babes University, Piaţa Eftimie Murgu Nr. 2, 300041, Timişoara, Romania.
authors.forename: Andreea
authors.initials: A
authors.lastname: Dobrescu
authors.affiliation: RTI-International, 3040 East Cornwallis Road, Research Triangle Park, NC 27709-2194, USA.
authors.forename: Tammeka Swinson
authors.initials: TS
authors.lastname: Evans
authors.affiliation: Department for Evidence-based Medicine and Clinical Epidemiology, Danube University, Dr. Karl Dorrek Strasse 30, 3500 Krems, Austria.
authors.forename: Kylie
authors.initials: K
authors.lastname: Thaler
authors.affiliation: Department for Evidence-based Medicine and Clinical Epidemiology, Danube University, Dr. Karl Dorrek Strasse 30, 3500 Krems, Austria.
authors.forename: Barbara
authors.initials: B
authors.lastname: Nussbaumer
authors.affiliation: Department for Evidence-based Medicine and Clinical Epidemiology, Danube University, Dr. Karl Dorrek Strasse 30, 3500 Krems, Austria.
authors.forename: Isolde
authors.initials: I
authors.lastname: Sommer
authors.affiliation: RTI-International, 3040 East Cornwallis Road, Research Triangle Park, NC 27709-2194, USA.
authors.forename: Kathleen N
authors.initials: KN
authors.lastname: Lohr
mesh_terms: D002986:Clinical Trials as Topic; D016021:Epidemiologic Studies; D019317:Evidence-Based Medicine; D006801:Humans; D013223:Statistics as Topic / Q000592:standards*
keywords: Effect estimates; First trial; GRADE; Quality of evidence; Systematic reviews; stability of treatment effects