title: Titian: Data Provenance Support in Spark.
abstract: Debugging data processing logic in Data-Intensive Scalable Computing (DISC) systems is a difficult and time consuming effort. Today's DISC systems offer very little tooling for debugging programs, and as a result programmers spend countless hours collecting evidence (e.g.,from log files) and performing trial and error debugging. To aid this effort, we builtTitian, a library that enablesdata provenance-tracking data through transformations-in Apache Spark. Data scientists using the Titian Spark extension will be able to quickly identify the input data at the root cause of a potential bug or outlier result. Titian is built directly into the Spark platform and offers data provenance support at interactive speeds-orders-of-magnitude faster than alternative solutions-while minimally impacting Spark job performance; observed overheads for capturing data lineage rarely exceed 30% above the baseline job execution time.
authors.affiliation: University of California, Los Angeles.
authors.forename: Matteo
authors.initials: M
authors.lastname: Interlandi
authors.affiliation: University of California, Los Angeles.
authors.forename: Kshitij
authors.initials: K
authors.lastname: Shah
authors.affiliation: University of California, Los Angeles.
authors.forename: Sai Deep
authors.initials: SD
authors.lastname: Tetali
authors.affiliation: University of California, Los Angeles.
authors.forename: Muhammad Ali
authors.initials: MA
authors.lastname: Gulzar
authors.affiliation: University of California, Los Angeles.
authors.forename: Seunghyun
authors.initials: S
authors.lastname: Yoo
authors.affiliation: University of California, Los Angeles.
authors.forename: Miryung
authors.initials: M
authors.lastname: Kim
authors.affiliation: University of California, Los Angeles.
authors.forename: Todd
authors.initials: T
authors.lastname: Millstein
authors.affiliation: University of California, Los Angeles.
authors.forename: Tyson
authors.initials: T
authors.lastname: Condie