Bias in trials comparing paired continuous tests can cause researchers to choose the wrong screening modality.
BACKGROUND
To compare the diagnostic accuracy of two continuous screening tests, a common approach is to test the difference between the areas under the receiver operating characteristic (ROC) curves. After study participants are screened with both screening tests, the disease status is determined as accurately as possible, either by an invasive, sensitive and specific secondary test, or by a less invasive, but less sensitive approach. For most participants, disease status is approximated through the less sensitive approach. The invasive test must be limited to the fraction of the participants whose results on either or both screening tests exceed a threshold of suspicion, or who develop signs and symptoms of the disease after the initial screening tests. The limitations of this study design lead to a bias in the ROC curves we call paired screening trial bias. This bias reflects the synergistic effects of inappropriate reference standard bias, differential verification bias, and partial verification bias. The absence of a gold reference standard leads to inappropriate reference standard bias. When different reference standards are used to ascertain disease status, it creates differential verification bias. When only suspicious screening test scores trigger a sensitive and specific secondary test, the result is a form of partial verification bias.


METHODS
For paired screening tests with bivariate normally distributed scores, we give formulae and programs to quantify the effect of paired screening trial bias on a paired comparison of area under the curves. We fix the prevalence of disease, and the chance a diseased subject manifests signs and symptoms. We derive the formulas for true sensitivity and specificity, and those for the sensitivity and specificity observed by the study investigator.


RESULTS
The observed area under the ROC curves is quite different from the true area under the ROC curves. The typical direction of the bias is a strong inflation in sensitivity, paired with a concomitant slight deflation of specificity.


CONCLUSIONS
In paired trials of screening tests, when area under the ROC curve is used as the metric, bias may lead researchers to make the wrong decision as to which screening test is better.
Glueck
Deborah H
DH
Department of Biostatistics, Colorado School of Public Health, University of Colorado, Denver, Aurora, CO, USA. Deborah.Glueck@ucdenver.edu
Lamb
Molly M
MM
O'Donnell
Colin I
CI
Ringham
Brandy M
BM
Brinton
John T
JT
Muller
Keith E
KE
Lewin
John M
JM
Alonzo
Todd A
TA
Pisano
Etta D
ED
D019540:Area Under Curve; D015982:Bias; D019411:Clinical Laboratory Techniques; D003661:Decision Support Techniques; D003951:Diagnostic Errors; D055088:Early Detection of Cancer; D006801:Humans; D008403:Mass Screening / Q000379:methods*; D015233:Models, Statistical; D012372:ROC Curve; D012108:Research Personnel; D012680:Sensitivity and Specificity