Statistical evaluation of the Predictive Toxicology Challenge 2000-2001.
BACKGROUND
The development of in silico models to predict chemical carcinogenesis from molecular structure would help greatly to prevent environmentally caused cancers. The Predictive Toxicology Challenge (PTC) competition was organized to test the state-of-the-art in applying machine learning to form such predictive models.


RESULTS
Fourteen machine learning groups generated 111 models. The use of Receiver Operating Characteristic (ROC) space allowed the models to be uniformly compared regardless of the error cost function. We developed a statistical method to test if a model performs significantly better than random in ROC space. Using this test as criteria five models performed better than random guessing at a significance level p of 0.05 (not corrected for multiple testing). Statistically the best predictor was the Viniti model for female mice, with p value below 0.002. The toxicologically most interesting models were Leuven2 for male mice, and Kwansei for female rats. These models performed well in the statistical analysis and they are in the middle of ROC space, i.e. distant from extreme cost assumptions. These predictive models were also independently judged by domain experts to be among the three most interesting, and are believed to include a small but significant amount of empirically learned toxicological knowledge.


BACKGROUND
PTC details and data can be found at: http://www.predictive-toxicology.org/ptc/.
Toivonen
Hannu
H
Department of Computer Science, PO Box 26 (Teollisuuskatu 23), FIN-00014 University of Helsinki, Finland. hannu.toivonen@cs.helsinki.fi
Srinivasan
Ashwin
A
King
Ross D
RD
Kramer
Stefan
S
Helma
Christoph
C
D000465:Algorithms; D000818:Animals; D001185:Artificial Intelligence; D015197:Carcinogenicity Tests / Q000379:methods*; D002273:Carcinogens / Q000737:chemistry* / Q000633:toxicity*; D003625:Data Collection; D016208:Databases, Factual; D004781:Environmental Exposure / Q000009:adverse effects; D005260:Female; D006078:Government Programs / Q000458:organization & administration; D008297:Male; D051379:Mice; D008954:Models, Biological; D015233:Models, Statistical; D009369:Neoplasms / Q000139:chemically induced*; D051381:Rats; D015203:Reproducibility of Results; D018570:Risk Assessment / Q000379:methods*; D012680:Sensitivity and Specificity; D012737:Sex Factors; D013045:Species Specificity; D013329:Structure-Activity Relationship; D014116:Toxicology / Q000379:methods; D014481:United States
D002273:Carcinogens