Benchmarking ontologies: bigger or better?
A scientific ontology is a formal representation of knowledge within a domain, typically including central concepts, their properties, and relations. With the rise of computers and high-throughput data collection, ontologies have become essential to data mining and sharing across communities in the biomedical sciences. Powerful approaches exist for testing the internal consistency of an ontology, but not for assessing the fidelity of its domain representation. We introduce a family of metrics that describe the breadth and depth with which an ontology represents its knowledge domain. We then test these metrics using (1) four of the most common medical ontologies with respect to a corpus of medical documents and (2) seven of the most popular English thesauri with respect to three corpora that sample language from medicine, news, and novels. Here we show that our approach captures the quality of ontological representation and guides efforts to narrow the breach between ontology and collective discourse within a domain. Our results also demonstrate key features of medical ontologies, English thesauri, and discourse from different domains. Medical ontologies have a small intersection, as do English thesauri. Moreover, dialects characteristic of distinct domains vary strikingly as many of the same words are used quite differently in medicine, news, and novels. As ontologies are intended to mirror the state of knowledge, our methods to tighten the fit between ontology and domain will increase their relevance for new areas of biomedical science and improve the accuracy and power of inferences computed across them.
Department of Biomedical Informatics, Columbia University, New York, New York, USA.
Lixia
L
Yao
Anna
A
Divoli
Ilya
I
Mayzus
James A
JA
Evans
Andrey
A
Rzhetsky
D016247:Information Storage and Retrieval
Proc AMIA Symp. 2001;:17-21
11825149
Nucleic Acids Res. 2008 Jan;36(Database issue):D724-8
18158299
AMIA Annu Symp Proc. 2008 Nov 06;:51-5
18999311
J Biomed Inform. 2006 Jun;39(3):288-98
16293444
Comput Biol Med. 2006 Jul-Aug;36(7-8):674-93
16144698
AMIA Annu Symp Proc. 2006;:121-5
17238315
Bull Med Libr Assoc. 1963 Jan;51:114-6
13982385
Wiley Interdiscip Rev Syst Biol Med. 2009 Nov-Dec;1(3):390-399
20052305
Ann Intern Med. 1999 Jul 20;131(2):117-26
10419428
Oecologia. 2009 Feb;159(1):15-26
18953572
J AHIMA. 1996 Jan;68(1):38
10163160
Proc Annu Symp Comput Appl Med Care. 1991;:194-8
1807586
Conf Proc IEEE Eng Med Biol Soc. 2004;2004:5415-8
17271570
Nucleic Acids Res. 2009 Jan;37(Database issue):D712-9
18981050
Bull Med Libr Assoc. 2000 Jul;88(3):265-6
10928714
Protein Sci. 2006 Jun;15(6):1544-9
16672243
Proc AMIA Annu Fall Symp. 1997;:485-9
9357673
J Am Med Inform Assoc. 1998 Jan-Feb;5(1):129-30
9471340
J Biomed Inform. 2004 Apr;37(2):108-19
15120657
J Am Med Inform Assoc. 2009 May-Jun;16(3):395-9
19261945
J Biomed Inform. 2006 Jun;39(3):267-73
16243005
Nat Genet. 2000 May;25(1):25-9
10802651
Bioinformatics. 2000 Mar;16(3):269-85
10869020
BMC Med Inform Decis Mak. 2008 Oct 27;8 Suppl 1:S2
19007439
PLoS One. 2008 Sep 09;3(9):e3158
18779866
BMC Bioinformatics. 2008 Apr 11;9 Suppl 3:S7
18426552
AMIA Annu Symp Proc. 2005;:639-43
16779118
Bioinformatics. 2009 Jun 15;25(12):i77-84
19478020
Int J Med Inform. 2009 Apr;78 Suppl 1:S86-94
18789754
Proc Natl Acad Sci U S A. 2006 Mar 28;103(13):4940-5
16543380
Proc AMIA Annu Fall Symp. 1996;:164-8
8947649
Bull Med Libr Assoc. 1993 Apr;81(2):184-94
8472004
J Am Med Inform Assoc. 2010 May-Jun;17(3):229-36
20442139