title: A threshold analysis assessed the credibility of conclusions from network meta-analysis.
abstract: To assess the reliability of treatment recommendations based on network meta-analysis (NMA).
  
  
  
  We consider evidence in an NMA to be potentially biased. Taking each pairwise contrast in turn, we use a structured series of threshold analyses to ask: (1) "How large would the bias in this evidence base have to be before it changed our decision?" and (2) "If the decision changed, what is the new recommendation?" We illustrate the method via two NMAs in which a Grading of Recommendations Assessment, Development and Evaluation (GRADE) assessment for NMAs has been implemented: weight loss and osteoporosis.
  
  
  
  Four of the weight-loss NMA estimates were assessed as "low" and six as "moderate" quality by GRADE; for osteoporosis, six were "low," nine were "moderate," and 1 was "high." The threshold analysis suggests plausible bias in 3 of 10 estimates in the weight-loss network could have changed the treatment recommendation. For osteoporosis, plausible bias in 6 of 16 estimates could change the recommendation. There was no relation between plausible bias changing a treatment recommendation and the original GRADE assessments.
  
  
  
  Reliability judgments on individual NMA contrasts do not help decision makers understand whether a treatment recommendation is reliable. Threshold analysis reveals whether the final recommendation is robust against plausible degrees of bias in the data.
authors.affiliation: School of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK. Electronic address: d.m.caldwell@bristol.ac.uk.
authors.forename: Deborah M
authors.initials: DM
authors.lastname: Caldwell
authors.affiliation: School of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK.
authors.forename: A E
authors.initials: AE
authors.lastname: Ades
authors.affiliation: School of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK.
authors.forename: Sofia
authors.initials: S
authors.lastname: Dias
authors.affiliation: School of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK.
authors.forename: Sarah
authors.initials: S
authors.lastname: Watkins
authors.affiliation: Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, 615 N Wolfe Street, Baltimore, MD 21205, USA.
authors.forename: Tianjing
authors.initials: T
authors.lastname: Li
authors.affiliation: Centre for Clinical Practice, National Institute for Health and Care Excellence, 10 Spring Gardens, London SW1A 2BU, UK.
authors.forename: Nichole
authors.initials: N
authors.lastname: Taske
authors.affiliation: Centre for Clinical Practice, National Institute for Health and Care Excellence, 10 Spring Gardens, London SW1A 2BU, UK.
authors.forename: Bhash
authors.initials: B
authors.lastname: Naidoo
authors.affiliation: School of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK.
authors.forename: Nicky J
authors.initials: NJ
authors.lastname: Welton
mesh_terms: D015982:Bias; D057186:Comparative Effectiveness Research / Q000706:statistics & numerical data; D006801:Humans; D000071076:Network Meta-Analysis as Topic; D010024:Osteoporosis / Q000628:therapy*; D017063:Outcome Assessment, Health Care / Q000706:statistics & numerical data*; D015203:Reproducibility of Results; D061217:Weight Reduction Programs / Q000706:statistics & numerical data*
keywords: Bias; Comparative effectiveness; GRADE; Health technology assessment; Mixed treatment comparison; Quality assessment; Reliability