title: Combining test statistics and models in bootstrapped model rejection: it is a balancing act.
abstract: BACKGROUND
  Model rejections lie at the heart of systems biology, since they provide conclusive statements: that the corresponding mechanistic assumptions do not serve as valid explanations for the experimental data. Rejections are usually done using e.g. the chi-square test (χ2) or the Durbin-Watson test (DW). Analytical formulas for the corresponding distributions rely on assumptions that typically are not fulfilled. This problem is partly alleviated by the usage of bootstrapping, a computationally heavy approach to calculate an empirical distribution. Bootstrapping also allows for a natural extension to estimation of joint distributions, but this feature has so far been little exploited.
  
  
  RESULTS
  We herein show that simplistic combinations of bootstrapped tests, like the max or min of the individual p-values, give inconsistent, i.e. overly conservative or liberal, results. A new two-dimensional (2D) approach based on parametric bootstrapping, on the other hand, is found both consistent and with a higher power than the individual tests, when tested on static and dynamic examples where the truth is known. In the same examples, the most superior test is a 2D χ2vsχ2, where the second χ2-value comes from an additional help model, and its ability to describe bootstraps from the tested model. This superiority is lost if the help model is too simple, or too flexible. If a useful help model is found, the most powerful approach is the bootstrapped log-likelihood ratio (LHR). We show that this is because the LHR is one-dimensional, because the second dimension comes at a cost, and because LHR has retained most of the crucial information in the 2D distribution. These approaches statistically resolve a previously published rejection example for the first time.
  
  
  CONCLUSIONS
  We have shown how to, and how not to, combine tests in a bootstrap setting, when the combination is advantageous, and when it is advantageous to include a second model. These results also provide a deeper insight into the original motivation for formulating the LHR, for the more general setting of nonlinear and non-nested models. These insights are valuable in cases when accuracy and power, rather than computational speed, are prioritized.
authors.forename: Rikard
authors.initials: R
authors.lastname: Johansson
authors.forename: Peter
authors.initials: P
authors.lastname: Strålfors
authors.affiliation: Department of Biomedical Engineering (IMT), Linköping University, Linköping, Sweden. gunnar.cedersund@liu.se.
authors.forename: Gunnar
authors.initials: G
authors.lastname: Cedersund
mesh_terms: D007328:Insulin / Q000378:metabolism; D016013:Likelihood Functions; D008954:Models, Biological; D015398:Signal Transduction; D049490:Systems Biology / Q000379:methods*
chemical_list: D007328:Insulin
references.citation: J R Soc Interface. 2009 Feb 6;6(31):187-202
references.pmid: 19205079
references.citation: BMC Syst Biol. 2012 Aug 05;6:95
references.pmid: 22863152
references.citation: Bioinformatics. 2009 Aug 1;25(15):1923-9
references.pmid: 19505944
references.citation: J Biol Chem. 2011 Jul 22;286(29):26028-41
references.pmid: 21572040
references.citation: Evolution. 1985 Jul;39(4):783-791
references.pmid: 28561359
references.citation: BMC Syst Biol. 2010 Apr 01;4:38
references.pmid: 20356406
references.citation: Nature. 2002 Nov 14;420(6912):206-10
references.pmid: 12432404
references.citation: Proc Natl Acad Sci U S A. 2001 Jul 31;98(16):8961-5
references.pmid: 11470909
references.citation: Bioinformatics. 1998;14(1):48-54
references.pmid: 9520501
references.citation: Bioinformatics. 2006 Feb 15;22(4):514-5
references.pmid: 16317076
references.citation: Bioinformatics. 2009 May 15;25(10):1300-6
references.pmid: 19289448
references.citation: Bioinformatics. 2008 Mar 15;24(6):833-9
references.pmid: 18057018
references.citation: Psychol Bull. 1951 May;48(3):156-8
references.pmid: 14834286
references.citation: BMC Syst Biol. 2009 Oct 31;3:105
references.pmid: 19878602
references.citation: FEBS J. 2012 Sep;279(18):3513-27
references.pmid: 22846178
references.citation: Biometrics. 1970 Mar;26(1):23-32
references.pmid: 5437360
references.citation: Mol Biol Evol. 2002 Jul;19(7):1171-80
references.pmid: 12082136
references.citation: Syst Biol. 2008 Oct;57(5):708-18
references.pmid: 18853358
references.citation: Proc Natl Acad Sci U S A. 1996 Jul 9;93(14):7085-90
references.pmid: 8692949
references.citation: FEBS J. 2011 Aug;278(16):2767-857
references.pmid: 21707921
references.citation: J Biol Chem. 2010 Jun 25;285(26):20171-9
references.pmid: 20421297
references.citation: FEBS J. 2009 Feb;276(4):903-22
references.pmid: 19215297
references.citation: PLoS Comput Biol. 2008 Feb;4(2):e30
references.pmid: 18282085
references.citation: Sci Signal. 2010 Mar 16;3(113):ra20
references.pmid: 20234003
references.citation: FEBS J. 2012 Mar;279(6):987-99
references.pmid: 22248283
references.citation: PLoS Comput Biol. 2008 Jun 20;4(6):e1000096
references.pmid: 18551197