title: BioBenchmark Toyama 2012: an evaluation of the performance of triple stores on biological data.
abstract: BACKGROUND
  Biological databases vary enormously in size and data complexity, from small databases that contain a few million Resource Description Framework (RDF) triples to large databases that contain billions of triples. In this paper, we evaluate whether RDF native stores can be used to meet the needs of a biological database provider. Prior evaluations have used synthetic data with a limited database size. For example, the largest BSBM benchmark uses 1 billion synthetic e-commerce knowledge RDF triples on a single node. However, real world biological data differs from the simple synthetic data much. It is difficult to determine whether the synthetic e-commerce data is efficient enough to represent biological databases. Therefore, for this evaluation, we used five real data sets from biological databases.
  
  
  RESULTS
  We evaluated five triple stores, 4store, Bigdata, Mulgara, Virtuoso, and OWLIM-SE, with five biological data sets, Cell Cycle Ontology, Allie, PDBj, UniProt, and DDBJ, ranging in size from approximately 10 million to 8 billion triples. For each database, we loaded all the data into our single node and prepared the database for use in a classical data warehouse scenario. Then, we ran a series of SPARQL queries against each endpoint and recorded the execution time and the accuracy of the query response.
  
  
  CONCLUSIONS
  Our paper shows that with appropriate configuration Virtuoso and OWLIM-SE can satisfy the basic requirements to load and query biological data less than 8 billion or so on a single node, for the simultaneous access of 64 clients. OWLIM-SE performs best for databases with approximately 11 million triples; For data sets that contain 94 million and 590 million triples, OWLIM-SE and Virtuoso perform best. They do not show overwhelming advantage over each other; For data over 4 billion Virtuoso works best. 4store performs well on small data sets with limited features when the number of triples is less than 100 million, and our test shows its scalability is poor; Bigdata demonstrates average performance and is a good open source triple store for middle-sized (500 million or so) data set; Mulgara shows a little of fragility.
authors.affiliation: Database Center for Life Science, Research Organization of Information and Systems, 178-4-4 Wakashiba, Kashiwa, Chiba 277-0871, Japan.
authors.forename: Hongyan
authors.initials: H
authors.lastname: Wu
authors.affiliation: INTEC Inc, 1-3-3 Shinsuna, Koto-ku, Tokyo 136-8637, Japan.
authors.forename: Toyofumi
authors.initials: T
authors.lastname: Fujiwara
authors.affiliation: Database Center for Life Science, Research Organization of Information and Systems, 178-4-4 Wakashiba, Kashiwa, Chiba 277-0871, Japan.
authors.forename: Yasunori
authors.initials: Y
authors.lastname: Yamamoto
authors.affiliation: Swiss-Prot group, SIB Swiss Institute of Bioinformatics, CMU, 1 Michel Servet, 1211 Geneva 4, Switzerland.
authors.forename: Jerven
authors.initials: J
authors.lastname: Bolleman
authors.affiliation: Database Center for Life Science, Research Organization of Information and Systems, 178-4-4 Wakashiba, Kashiwa, Chiba 277-0871, Japan.
authors.forename: Atsuko
authors.initials: A
authors.lastname: Yamaguchi
references.citation: Nucleic Acids Res. 2008 Jan;36(Database issue):D402-8
references.pmid: 17984079
references.citation: Nucleic Acids Res. 2013 Jan;41(Database issue):D43-7
references.pmid: 23161681
references.citation: Database (Oxford). 2011 Apr 15;2011:bar013
references.pmid: 21498548
references.citation: BMC Bioinformatics. 2012 Jan 25;13 Suppl 1:S3
references.pmid: 22373359
references.citation: Nucleic Acids Res. 2012 Jan;40(Database issue):D445-52
references.pmid: 22110033
references.citation: Brief Bioinform. 2013 Jan;14(1):109-25
references.pmid: 22492191
references.citation: Science. 1990 Apr 6;248(4951):49-51
references.pmid: 2181666
references.citation: Brief Bioinform. 2009 Mar;10(2):177-92
references.pmid: 19304873
references.citation: Nucleic Acids Res. 2006 Jan 1;34(Database issue):D655-8
references.pmid: 16381952
references.citation: Nucleic Acids Res. 2012 Jan;40(Database issue):D453-60
references.pmid: 21976737
references.citation: Nucleic Acids Res. 1997 Jan 1;25(1):1-6
references.pmid: 9016491
references.citation: Nucleic Acids Res. 2012 Jan;40(Database issue):D38-42
references.pmid: 22110025
references.citation: Nucleic Acids Res. 2002 Jan 1;30(1):207-10
references.pmid: 11752295
references.citation: Brief Bioinform. 2009 Jul;10(4):392-407
references.pmid: 19457869
references.citation: Nucleic Acids Res. 2003 Jan 1;31(1):248-50
references.pmid: 12519993
references.citation: Nat Neurosci. 2004 May;7(5):467-72
references.pmid: 15114360
references.citation: Nucleic Acids Res. 2013 Jan;41(Database issue):D475-82
references.pmid: 23193259
references.citation: Nucleic Acids Res. 2003 Jan 1;31(1):94-6
references.pmid: 12519956
references.citation: J Biomed Semantics. 2013 Feb 11;4(1):6
references.pmid: 23398680