title: Optimizing Interactive Development of Data-Intensive Applications.
abstract: Modern Data-Intensive Scalable Computing (DISC) systems are designed to process data through batch jobs that execute programs (e.g., queries) compiled from a high-level language. These programs are often developed interactively by posing ad-hoc queries over the base data until a desired result is generated. We observe that there can be significant overlap in the structure of these queries used to derive the final program. Yet, each successive execution of a slightly modified query is performed anew, which can significantly increase the development cycle. Vega is an Apache Spark framework that we have implemented for optimizing a series of similar Spark programs, likely originating from a development or exploratory data analysis session. Spark developers (e.g., data scientists) can leverage Vega to significantly reduce the amount of time it takes to re-execute a modified Spark program, reducing the overall time to market for their Big Data applications.
authors.affiliation: University of California, Los Angeles.
authors.forename: Matteo
authors.initials: M
authors.lastname: Interlandi
authors.affiliation: University of California, Los Angeles.
authors.forename: Sai Deep
authors.initials: SD
authors.lastname: Tetali
authors.affiliation: University of California, Los Angeles.
authors.forename: Muhammad Ali
authors.initials: MA
authors.lastname: Gulzar
authors.affiliation: University of California, Los Angeles.
authors.forename: Joseph
authors.initials: J
authors.lastname: Noor
authors.affiliation: University of California, Los Angeles.
authors.forename: Tyson
authors.initials: T
authors.lastname: Condie
authors.affiliation: University of California, Los Angeles.
authors.forename: Miryung
authors.initials: M
authors.lastname: Kim
authors.affiliation: University of California, Los Angeles.
authors.forename: Todd
authors.initials: T
authors.lastname: Millstein
keywords: Big Data; H.2.4 [Information Systems]: Database Management—query processing; Incremental Evaluation; Interactive Development; Languages; Performance; Query Rewriting; Spark; Theory; parallel databases