Hi Eron, Great to see so much progress on the Mesos implementation! Thank you for sharing the code with us.
I'm not entirely sure whether we actually wait on the completion of FLIP-6. We might complete the Mesos support for the 1.2.0 release and port its code to the new RPC abstraction that comes with FLIP-6 afterwards. That is still to decide. It is probably also acceptable to have the new RPC layer and still use direct Akka code for Mesos. Client changes should be relatively independent of FLIP-6 and be made in such way that they are easily portable to FLIP-6. All in all, we are probably best of with merging your changes to the master quickly since Mesos support will be one of the most important new features of Flink 1.2.0. Best, Max On Mon, Sep 26, 2016 at 11:37 PM, Wright, Eron <ewri...@live.com> wrote: > > Hello, > > The code I presented at FF2016 represents a 'status-quo' approach to > realizing a specific scenario - "mesos-session.sh". But the final solution > will involve CLI changes and the full realization of a dispatcher, which > conflicts with FLIP-6. We should advance the client/dispatcher design of > FLIP-6 before I make further changes. Also we must decide whether the > Mesos work should take FLIP-6 as a dependency, and/or strive to land a > useable subset into master. > > Here's a summary of the code I presented, as reference for ongoing FLIP-6 > design discussion. A fresh PR for convenience: > https://github.com/EronWright/flink/pull/1/files > > 1. MesosDispatcher (Backend). The 'backend' acts as a Mesos framework, to > launch an AppMaster as a Mesos task. For each session, the backend accepts > "session parameters" which define the libraries, configuration, resource > profile and other parameters needed to launch the AppMaster. The backend > persists the information for recovery purposes. Leader election is also > used. Not included is a dispatcher 'frontend' - a formalized API surface or > REST server. > > 2. SessionParameters. Captures the information needed to launch a session > based on an AppMaster. A session is a stateful execution environment for a > program. The session parameters can also be understood as the historical > inputs to yarn-session.sh, plus job inputs per FLIP-6. I acknowledge that > the term 'session' is a working term. > > 3 . MesosDispatcherRunner. The runner for the 'remote dispatcher' scenario, > which would be started by Marathon, expose a REST API with which to > submit/manage jobs, and host the above dispatcher backend. > > 4. FlinkMesosSessionCli. This class mirrors the FlinkYarnSessionCli, which > is used in both the 'flink run' scenario and 'yarn-session' scenario. I > didn't fully implement the CustomCommandLine interface which yields a > ClusterClient, because the dispatcher API must be fleshed out first. > > 5. SessionArtifactHelper. An attempt to consolidate logic related to > session artifacts (i.e. ship files). > > 6. DispatcherClient. The client interface for the dispatcher. In > concept there could be numerous implementations - a > 'remote' impl which would make REST calls to a remote dispatcher, a 'local' > impl which would host the dispatcher directly. Seen in this PR is only the > latter but it might be throw-away code. > > 7. LaunchableMesosSession. This class generates the effective container > environment at launch time. > > 8. ContaineredJobMasterParameters. Refactored from YARN code for sharing > purposes. > > -Eron
