Pitfalls in retrospective analyses of biomarkers: a case study with metastatic melanoma patients.
BACKGROUND
Reliable prognostic biomarkers of survival and response to treatment are clearly important in oncology, and many studies have been carried out with the objective of identifying new prognostic biomarkers. Retrospective analysis of blood banked from patients is a frequently used paradigm for these studies. We describe a new study of the association of serum biomarker level with overall survival in melanoma patients, and the problems encountered in carrying it out.


METHODS
Blood samples from 56 patients with stage IV metastatic melanoma were drawn prior to initiation of any treatment for their disease. Sera from the samples were stored for up to 94 months at -80°C, and were subsequently thawed at the same time and tested by multiplex Luminex assay for 30 analytes (cytokines, chemokines and growth factors). Cox regression analysis was used to assess the association between these analytes and time-to-death.


RESULTS
Of the 30 analytes, 17 were associated with survival, most strongly so, and in all cases, a higher analyte level was associated with increased survival. In addition, the correlations of the levels of all possible pairs of analytes were all positive and in almost all cases highly significant. However, these results are artifacts that arise from the combination of two peculiarities of the data: the apparent decrease in analyte level with storage time, and the uniformly shorter storage times of the samples from censored patients than the storage times of the samples from patients who died.


CONCLUSIONS
All retrospective studies can have hidden biases, and thus investigators should not claim new findings before examining the data in detail with the goal of determining whether the findings could be spurious. There were several suspicious findings in our initial analyses: too many analytes found significant, too many very small p-values, a uniformly positive association of analyte level with survival, and a uniformly positive correlation between analyte levels. We were convinced that these findings must be artifacts, and further analyses showed that the findings could be explained by an apparent decrease of analyte level storage time.
Biostatistics Facility, University of Pittsburgh Cancer Institute, 201 North Craig Street, Suite 325, Pittsburgh, PA 15213, United States. dmp71@pitt.edu
Douglas M
DM
Potter
Lisa H
LH
Butterfield
Sherrie J
SJ
Divito
Cindy A
CA
Sander
John M
JM
Kirkwood
D000328:Adult; D000368:Aged; D000369:Aged, 80 and over; D014408:Biomarkers, Tumor / Q000097:blood*; D005260:Female; D006801:Humans; D008297:Male; D008545:Melanoma / Q000097:blood*; D008875:Middle Aged; D016016:Proportional Hazards Models; D012189:Retrospective Studies; D016019:Survival Analysis; D055815:Young Adult
D014408:Biomarkers, Tumor