Exploratory data analysis using set operations and ordinal mapping.
Exploratory data analysis requires the ability to issue ad hoc queries to filter and summarise data sets. As the sizes of health data sets grow, traditional methods of processing data have difficulty in providing acceptable response times for such queries. An alternative method is described which combines complete vertical partitioning of data with set operations on ordinal mappings (SOOM). An initial implementation of the technique provides significantly better performance than a conventional SQL database on typical exploratory data analysis queries. The use of parallel, distributed computation to further increase the performance of the technique appears to be feasible.
Epidemiology and Surveillance Branch, New South Wales Department of Health, Locked Mail Bag 961, North Sydney, NSW 2059, Australia. tchur@doh.health.nsw.gov
Tim
T
Churches
D000465:Algorithms; D003205:Computing Methodologies; D003628:Database Management Systems; D016206:Databases, Bibliographic; D004812:Epidemiologic Methods; D005240:Feasibility Studies; D016247:Information Storage and Retrieval / Q000379:methods*; D012984:Software; D013223:Statistics as Topic / Q000379:methods*; D014584:User-Computer Interface