title: Evaluating and reducing the effect of data corruption when applying bag of words approaches to medical records.
abstract: Unlike journal corpora, which are supposed to be carefully reviewed before being published, the quality of documents in a patient record are often corrupted by mispelled words and conventional graphies or abbreviations. After a survey of the domain, the paper focuses on evaluating the effect of such corruption on an information retrieval (IR) engine. The IR system uses a classical bag of words approach, with stems as representation items and term frequency-inverse document frequency (tf-idf) as weighting schema; we pay special attention to the normalization factor. First results shows that even low corruption levels (3%) do affect retrieval effectiveness (4-7%), whereas higher corruption levels can affect retrieval effectiveness by 25%. Then, we show that the use of an improved automatic spelling correction system, applied on the corrupted collection, can almost restore the retrieval effectiveness of the engine.
authors.affiliation: Theoretical Computer Science Laboratory, Swiss Federal Institute of Technology, Lausanne, Switzerland. patrick.ruch@epfl.ch
authors.forename: P
authors.initials: P
authors.lastname: Ruch
authors.forename: R
authors.initials: R
authors.lastname: Baud
authors.forename: A
authors.initials: A
authors.lastname: Geissbühler
mesh_terms: D005544:Forecasting; D006801:Humans; D016247:Information Storage and Retrieval; D008037:Linguistics; D016347:Medical Records Systems, Computerized; D009323:Natural Language Processing; D012984:Software; D018875:Vocabulary, Controlled