title: Consistency between direct and indirect trial evidence: is direct evidence always more reliable?
abstract: OBJECTIVE
  To present a case study involving the reduction in incidence of febrile neutropenia (FN) after chemotherapy with granulocyte colony-stimulating factors (G-CSFs), illustrating difficulties that may arise when following the common preference for direct evidence over indirect evidence.
  
  
  METHODS
  Evidence of the efficacy of treatments was identified from two previous systematic reviews. We used Bayesian evidence synthesis to estimate relative treatment effects based on direct evidence, indirect evidence, and both pooled together. We checked for inconsistency between direct and indirect evidence and explored the role of one specific trial using cross-validation. A subsequent review identified further studies not available at the time of the original analysis. We repeated the analyses on the enlarged evidence base.
  
  
  RESULTS
  We found substantial inconsistency in the original evidence base. The median odds ratio of FN for primary pegfilgrastim versus no primary G-CSF was 0.06 (95% credible interval: 0.02-0.19) based on direct evidence, but 0.27 (95% credible interval: 0.13-0.53) based on indirect evidence (P value for consistency hypothesis 0.027). The additional trials were consistent with the earlier indirect, rather than the direct, evidence, and there was no inconsistency between direct and indirect estimates in the updated evidence. The earlier inconsistency was due to one trial comparing primary pegfilgrastim with no primary G-CSF. Predictive cross-validation showed that this study was inconsistent with the evidence as a whole and with other trials making this comparison.
  
  
  CONCLUSIONS
  Both the Cochrane Handbook and the NICE Methods Guide express a preference for direct evidence. A more robust strategy, which is in line with the accepted principles of evidence synthesis, would be to combine all relevant and appropriate information, whether direct or indirect.
authors.affiliation: University of Bristol, Bristol, UK. jason.madan@bris.ac.uk
authors.forename: Jason
authors.initials: J
authors.lastname: Madan
authors.forename: Matt D
authors.initials: MD
authors.lastname: Stevenson
authors.forename: Katy L
authors.initials: KL
authors.lastname: Cooper
authors.forename: A E
authors.initials: AE
authors.lastname: Ades
authors.forename: Sophie
authors.initials: S
authors.lastname: Whyte
authors.forename: Ron
authors.initials: R
authors.lastname: Akehurst
mesh_terms: D001499:Bayes Theorem; D000069585:Filgrastim; D016179:Granulocyte Colony-Stimulating Factor / Q000008:administration & dosage*; D006801:Humans; D000078224:Lenograstim; D009503:Neutropenia / Q000517:prevention & control*; D011092:Polyethylene Glycols; D011994:Recombinant Proteins / Q000008:administration & dosage; D015203:Reproducibility of Results; D012107:Research Design
chemical_list: D011994:Recombinant Proteins; D016179:Granulocyte Colony-Stimulating Factor; C455861:pegfilgrastim; D011092:Polyethylene Glycols; D000078224:Lenograstim; D000069585:Filgrastim