Validation of the summary ROC for diagnostic test meta-analysis: a Monte Carlo simulation.
OBJECTIVE
The author performed this study to test a technique for validating the logit regression method for summary receiver operating characteristic (ROC) meta-analysis, perform initial validation studies, and identify areas for further investigation.


METHODS
Monte Carlo simulation was performed by using a custom macro program for a personal computer spreadsheet. The program creates simulated data sets based on user-specified parameters, performs a meta-analysis on the data sets, and logs the results so the accuracy and variability of the method can be measured. The program can also be used to measure the effects of changes in study design and meta-analysis parameters.


RESULTS
For the base case of a small meta-analysis (10 studies) of small trials (mean, 50 patients), the meta-analysis results closely matched the input sensitivity and specificity when they were less than 80%. Systematic errors, if any, were small. At sensitivities and specificities greater than 80%, the true sensitivity or specificity was underestimated by up to 2% in the meta-analysis. Confidence intervals calculated with the summary ROC curve were reasonably conservative, although they too fell below the true results when sensitivity or specificity was greater than 80%. The underestimation was eliminated when the simulation was repeated for a much larger trial (mean, 1,000 cases per study)--even with a sensitivity and specificity of 98%.


CONCLUSIONS
The Littenberg-Moses method for summary ROC meta-analysis is effective for obtaining an accurate summary estimate of diagnostic test performance, although the continuity correction introduces a small downward bias in the meta-analysis of small trials.
Mitchell
Matthew D
MD
Technology Assessment Group, ECRI, 5200 Butler Pike, Plymouth Meeting, PA 19462, USA.
D003198:Computer Simulation; D006801:Humans; D016015:Logistic Models; D015201:Meta-Analysis as Topic; D009010:Monte Carlo Method; D012372:ROC Curve; D012680:Sensitivity and Specificity; D012984:Software