title: REEF: Retainable Evaluator Execution Framework.
abstract: Resource Managers like Apache YARN have emerged as a critical layer in the cloud computing system stack, but the developer abstractions for leasing cluster resources and instantiating application logic are very low-level. This flexibility comes at a high cost in terms of developer effort, as each application must repeatedly tackle the same challenges (e.g., fault-tolerance, task scheduling and coordination) and re-implement common mechanisms (e.g., caching, bulk-data transfers). This paper presents REEF, a development framework that provides a control-plane for scheduling and coordinating task-level (data-plane) work on cluster resources obtained from a Resource Manager. REEF provides mechanisms that facilitate resource re-use for data caching, and state management abstractions that greatly ease the development of elastic data processing work-flows on cloud platforms that support a Resource Manager service. REEF is being used to develop several commercial offerings such as the Azure Stream Analytics service. Furthermore, we demonstrate REEF development of a distributed shell application, a machine learning algorithm, and a port of the CORFU [4] system. REEF is also currently an Apache Incubator project that has attracted contributors from several instititutions.
authors.affiliation: Microsoft.
authors.forename: Markus
authors.initials: M
authors.lastname: Weimer
authors.affiliation: Microsoft.
authors.forename: Yingda
authors.initials: Y
authors.lastname: Chen
authors.affiliation: Seoul National University.
authors.forename: Byung-Gon
authors.initials: BG
authors.lastname: Chun
authors.affiliation: UCLA.
authors.forename: Tyson
authors.initials: T
authors.lastname: Condie
authors.affiliation: Microsoft.
authors.forename: Carlo
authors.initials: C
authors.lastname: Curino
authors.affiliation: Microsoft.
authors.forename: Chris
authors.initials: C
authors.lastname: Douglas
authors.affiliation: Seoul National University.
authors.forename: Yunseong
authors.initials: Y
authors.lastname: Lee
authors.affiliation: Microsoft.
authors.forename: Tony
authors.initials: T
authors.lastname: Majestro
authors.affiliation: VMware.
authors.forename: Dahlia
authors.initials: D
authors.lastname: Malkhi
authors.affiliation: Microsoft.
authors.forename: Sergiy
authors.initials: S
authors.lastname: Matusevych
authors.affiliation: University of Washington.
authors.forename: Brandon
authors.initials: B
authors.lastname: Myers
authors.affiliation: Microsoft.
authors.forename: Shravan
authors.initials: S
authors.lastname: Narayanamurthy
authors.affiliation: Microsoft.
authors.forename: Raghu
authors.initials: R
authors.lastname: Ramakrishnan
authors.affiliation: Microsoft.
authors.forename: Sriram
authors.initials: S
authors.lastname: Rao
authors.affiliation: Pure Storage.
authors.forename: Russell
authors.initials: R
authors.lastname: Sears
authors.affiliation: Microsoft.
authors.forename: Beysim
authors.initials: B
authors.lastname: Sezgin
authors.affiliation: Microsoft.
authors.forename: Julia
authors.initials: J
authors.lastname: Wang
keywords: Big Data; Database; Distributed Systems; High Performance Computing; Machine Learning