title: A new approach for cleansing geographical dataset using Levenshtein distance, prior knowledge and contextual information.
abstract: Epidemiological studies are necessary to take public health decisions. Their relevance depends on the quality of data. Doctors in continuous care collect a big amount of data that can be used for epidemiological purpose, but spatial data may be dirty; based on city names, the localization is imprecise, even more if it is misspelled. The only way to identify a city without ambiguity is to use its identifier, which can be retrieved by cleansing geographical textual data. In France, cities are organized in administrative zones called departments and some city names are shared by several cities in several departments. The clear identification of the department and the city name allows to deduce the city unique identifier and to make some spatial analysis such as epidemiological studies. In this paper, we propose a method to cleanse such data, using several steps. After having standardized the text to cleanse, we use the Levenshtein distance to generate a first set of propositions. Finally, the propositions are filtered, by removing the less likely candidates, so that it remains only one, which becomes the chosen city. Tested on a dataset of 9818 entries, we obtained 89.1% of concordance, whereas the standard Levenshtein distance obtained 70.5%. This demonstrates that our method has better results.
authors.affiliation: INSERM, U1142, LIMICS, F-75006, Paris, France; Sorbonne Universités, UPMC Univ Paris 06, UMR_S 1142, LIMICS, F-75006, Paris, France; Université Paris 13, Sorbonne Paris Cité, LIMICS, (UMR_S 1142), F-93430, Villetaneuse, France.
authors.forename: Adrien
authors.initials: A
authors.lastname: Ugon
authors.affiliation: INSERM, U1142, LIMICS, F-75006, Paris, France; Sorbonne Universités, UPMC Univ Paris 06, UMR_S 1142, LIMICS, F-75006, Paris, France; Université Paris 13, Sorbonne Paris Cité, LIMICS, (UMR_S 1142), F-93430, Villetaneuse, France.
authors.forename: Thomas
authors.initials: T
authors.lastname: Nicolas
authors.affiliation: INSERM, U1142, LIMICS, F-75006, Paris, France; Sorbonne Universités, UPMC Univ Paris 06, UMR_S 1142, LIMICS, F-75006, Paris, France; Université Paris 13, Sorbonne Paris Cité, LIMICS, (UMR_S 1142), F-93430, Villetaneuse, France.
authors.forename: Marion
authors.initials: M
authors.lastname: Richard
authors.affiliation: IRSAN, 37 rue des Mathurins, F-75008, Paris, France.
authors.forename: Patrick
authors.initials: P
authors.lastname: Guerin
authors.affiliation: IRSAN, 37 rue des Mathurins, F-75008, Paris, France.
authors.forename: Pascal
authors.initials: P
authors.lastname: Chansard
authors.affiliation: IRSAN, 37 rue des Mathurins, F-75008, Paris, France.
authors.forename: Christophe
authors.initials: C
authors.lastname: Demoor
authors.affiliation: INSERM, U1142, LIMICS, F-75006, Paris, France; Sorbonne Universités, UPMC Univ Paris 06, UMR_S 1142, LIMICS, F-75006, Paris, France; Université Paris 13, Sorbonne Paris Cité, LIMICS, (UMR_S 1142), F-93430, Villetaneuse, France.
authors.forename: Laurent
authors.initials: L
authors.lastname: Toubiana
mesh_terms: D000465:Algorithms; D000068598:Data Accuracy; D016208:Databases, Factual; D005602:France; D016247:Information Storage and Retrieval / Q000379:methods; D000069550:Machine Learning; D009323:Natural Language Processing; D010363:Pattern Recognition, Automated / Q000379:methods*; D012660:Semantics; D019371:Topography, Medical / Q000379:methods*