title: Efficient Record Linkage Algorithms Using Complete Linkage Clustering.
abstract: Data from different agencies share data of the same individuals. Linking these datasets to identify all the records belonging to the same individuals is a crucial and challenging problem, especially given the large volumes of data. A large number of available algorithms for record linkage are prone to either time inefficiency or low-accuracy in finding matches and non-matches among the records. In this paper we propose efficient as well as reliable sequential and parallel algorithms for the record linkage problem employing hierarchical clustering methods. We employ complete linkage hierarchical clustering algorithms to address this problem. In addition to hierarchical clustering, we also use two other techniques: elimination of duplicate records and blocking. Our algorithms use sorting as a sub-routine to identify identical copies of records. We have tested our algorithms on datasets with millions of synthetic records. Experimental results show that our algorithms achieve nearly 100% accuracy. Parallel implementations achieve almost linear speedups. Time complexities of these algorithms do not exceed those of previous best-known algorithms. Our proposed algorithms outperform previous best-known algorithms in terms of accuracy consuming reasonable run times.
authors.affiliation: Department of Computer Science and Engineering, University of Connecticut, Storrs, Connecticut, United States of America.
authors.forename: Abdullah-Al
authors.initials: AA
authors.lastname: Mamun
authors.affiliation: Institute for Public Health Research, University of Connecticut, East Hartford, Connecticut, United States of America.
authors.forename: Robert
authors.initials: R
authors.lastname: Aseltine
authors.affiliation: Department of Computer Science and Engineering, University of Connecticut, Storrs, Connecticut, United States of America.
authors.forename: Sanguthevar
authors.initials: S
authors.lastname: Rajasekaran
mesh_terms: D000465:Algorithms; D016000:Cluster Analysis; D005568:Forms and Records Control / Q000706:statistics & numerical data*; D006801:Humans; D008498:Medical Record Linkage / Q000379:methods*; D000068241:Personally Identifiable Information / Q000706:statistics & numerical data*; D011996:Records; D013997:Time Factors