The need to approximate the use-case in clinical machine learning.
The availability of smartphone and wearable sensor technology is leading to a rapid accumulation of human subject data, and machine learning is emerging as a technique to map those data into clinical predictions. As machine learning algorithms are increasingly used to support clinical decision making, it is vital to reliably quantify their prediction accuracy. Cross-validation (CV) is the standard approach where the accuracy of such algorithms is evaluated on part of the data the algorithm has not seen during training. However, for this procedure to be meaningful, the relationship between the training and the validation set should mimic the relationship between the training set and the dataset expected for the clinical use. Here we compared two popular CV methods: record-wise and subject-wise. While the subject-wise method mirrors the clinically relevant use-case scenario of diagnosis in newly recruited subjects, the record-wise strategy has no such interpretation. Using both a publicly available dataset and a simulation, we found that record-wise CV often massively overestimates the prediction accuracy of the algorithms. We also conducted a systematic review of the relevant literature, and found that this overly optimistic method was used by almost half of the retrieved studies that used accelerometers, wearable sensors, or smartphones to predict clinical outcomes. As we move towards an era of machine learning-based diagnosis and treatment, using proper methods to evaluate their accuracy is crucial, as inaccurate results can mislead both clinicians and data scientists.
Department of Preventive Medicine, Northwestern University, 10th floor, Rubloff Bldg, 750 N Lake Shore Dr, Chicago, IL 60611, USA.|Department of Physical Medicine and Rehabilitation, Northwestern University, 345 E Superior St, Suite 1479, Chicago, IL 60611, USA.
Sohrab
S
Saeb
Department of Physical Medicine and Rehabilitation, Northwestern University, 345 E Superior St, Suite 1479, Chicago, IL 60611, USA.|Shirley Ryan Ability Lab, Room 1401, 11th Floor, 355 E Erie St., Chicago, IL 60611, USA.
Luca
L
Lonini
Department of Physical Medicine and Rehabilitation, Northwestern University, 345 E Superior St, Suite 1479, Chicago, IL 60611, USA.|Shirley Ryan Ability Lab, Room 1401, 11th Floor, 355 E Erie St., Chicago, IL 60611, USA.
Arun
A
Jayaraman
Department of Preventive Medicine, Northwestern University, 10th floor, Rubloff Bldg, 750 N Lake Shore Dr, Chicago, IL 60611, USA.
David C
DC
Mohr
Department of Physical Medicine and Rehabilitation, Northwestern University, 345 E Superior St, Suite 1479, Chicago, IL 60611, USA.
Konrad P
KP
Kording
D061725:Accelerometry; D000465:Algorithms; D015444:Exercise; D006801:Humans; D000069550:Machine Learning; D018670:Monitoring, Ambulatory / Q000379:methods*; D015203:Reproducibility of Results; D000068997:Smartphone; D000076251:Wearable Electronic Devices
Machine learning; clinical outcomes; cross-validation; diagnosis; prediction accuracy; rehabilitation outcomes; smartphones; wearable technology