Assessing performance of prediction rules in machine learning.
BACKGROUND
An important goal in machine learning is to assess the degree to which prediction rules are robust and replicable, since these rules are used for decision making and for planning follow-up studies. This requires an estimate of a prediction rule's true error rate, a statistic that can be estimated by resampling data. However, there are many possible approaches depending upon whether we draw observations with or without replacement, or sample once, repeatedly, or not at all, and the pros and cons of each are often unclear. This study illustrates and compares different methods for estimating true error with the aim of providing practical guidance to users of machine learning techniques.


METHODS
We conducted Monte Carlo simulation studies using four different error estimators: bootstrap, split sample, resubstitution and a direct estimate of true error. Here, 'split sample' refers to a single random partition of the data into a pair of training and test samples, a popular scheme. We used stochastic gradient boosting as a learning algorithm, and considered data from two studies for which the underlying data mechanism was known to be complex: a library of 6000 tripeptide substrates collected for analysis of proteasome inhibition as part of anticancer drug design, and a cardiovascular study involving 600 subjects receiving antiplatelet treatment for acute coronary syndrome.


RESULTS
There were important differences in the performance of the various error estimators examined. Error estimators for split sample and resubstitution, while being the most transparent in action and the simplest to apply, did not quantify the performance of prediction rules as accurately as the bootstrap. This was true for both types of study data, despite their highly different nature.


CONCLUSIONS
The robustness and reliability of decisions based on analysis of genomics data could, in many cases, be improved by following best practices for prediction error estimation. For this, techniques such as bootstrap should be considered.
Martin
Rory
R
Millennium Pharmaceuticals, Cambridge MA 02139, USA. rory.martin.phd@gmail.com
Yu
Kai
K
D000465:Algorithms; D000970:Antineoplastic Agents / Q000494:pharmacology; D001185:Artificial Intelligence; D003327:Coronary Disease / Q000188:drug therapy; D006801:Humans; D066298:In Vitro Techniques; D009010:Monte Carlo Method; D009842:Oligopeptides / Q000494:pharmacology; D019151:Peptide Library; D010597:Pharmacogenetics / Q000706:statistics & numerical data*; D010975:Platelet Aggregation Inhibitors / Q000627:therapeutic use; D011480:Protease Inhibitors / Q000494:pharmacology; D061988:Proteasome Inhibitors
D000970:Antineoplastic Agents; D009842:Oligopeptides; D019151:Peptide Library; D010975:Platelet Aggregation Inhibitors; D011480:Protease Inhibitors; D061988:Proteasome Inhibitors