title: A Distributed Look-up Architecture for Text Mining Applications using MapReduce.
abstract: Text mining applications typically involve statistical models that require accessing and updating model parameters in an iterative fashion. With the growing size of the data, such models become extremely parameter rich, and naive parallel implementations fail to address the scalability problem of maintaining a distributed look-up table that maps model parameters to their values. We evaluate several existing alternatives to provide coordination among worker nodes in Hadoop [11] clusters, and suggest a new multi-layered look-up architecture that is specifically optimized for certain problem domains. Our solution exploits the power-law distribution characteristics of the phrase or n-gram counts in large corpora while utilizing a Bloom Filter [2], in-memory cache, and an HBase [12] cluster at varying levels of abstraction.
authors.affiliation: Department of Computer Science, University of Chicago.
authors.forename: Atilla Soner
authors.initials: AS
authors.lastname: Balkir
authors.affiliation: Department of Computer Science, University of Chicago, Argonne National Laboratory.
authors.forename: Ian
authors.initials: I
authors.lastname: Foster
authors.affiliation: Computation Institute, University of Chicago, Argonne National Laboratory.
authors.forename: Andrey
authors.initials: A
authors.lastname: Rzhetsky