title: BigDebug: Debugging Primitives for Interactive Big Data Processing in Spark.
abstract: Developers use cloud computing platforms to process a large quantity of data in parallel when developing big data analytics. Debugging the massive parallel computations that run in today's data-centers is time consuming and error-prone. To address this challenge, we design a set of interactive, real-time debugging primitives for big data processing in Apache Spark, the next generation data-intensive scalable cloud computing platform. This requires re-thinking the notion of step-through debugging in a traditional debugger such as gdb, because pausing the entire computation across distributed worker nodes causes significant delay and naively inspecting millions of records using a watchpoint is too time consuming for an end user. First, BIGDEBUG's simulated breakpoints and on-demand watchpoints allow users to selectively examine distributed, intermediate data on the cloud with little overhead. Second, a user can also pinpoint a crash-inducing record and selectively resume relevant sub-computations after a quick fix. Third, a user can determine the root causes of errors (or delays) at the level of individual records through a fine-grained data provenance capability. Our evaluation shows that BIGDEBUG scales to terabytes and its record-level tracing incurs less than 25% overhead on average. It determines crash culprits orders of magnitude more accurately and provides up to 100% time saving compared to the baseline replay debugger. The results show that BIGDEBUG supports debugging at interactive speeds with minimal performance impact.
authors.affiliation: University of California, Los Angeles.
authors.forename: Muhammad Ali
authors.initials: MA
authors.lastname: Gulzar
authors.affiliation: University of California, Los Angeles.
authors.forename: Matteo
authors.initials: M
authors.lastname: Interlandi
authors.affiliation: University of California, Los Angeles.
authors.forename: Seunghyun
authors.initials: S
authors.lastname: Yoo
authors.affiliation: University of California, Los Angeles.
authors.forename: Sai Deep
authors.initials: SD
authors.lastname: Tetali
authors.affiliation: University of California, Los Angeles.
authors.forename: Tyson
authors.initials: T
authors.lastname: Condie
authors.affiliation: University of California, Los Angeles.
authors.forename: Todd
authors.initials: T
authors.lastname: Millstein
authors.affiliation: University of California, Los Angeles.
authors.forename: Miryung
authors.initials: M
authors.lastname: Kim
keywords: Debugging; big data analytics; data-intensive scalable computing (DISC); fault localization and recovery; interactive tools