Hello, The code I presented at FF2016 represents a 'status-quo' approach to realizing a specific scenario - "mesos-session.sh". But the final solution will involve CLI changes and the full realization of a dispatcher, which conflicts with FLIP-6. We should advance the client/dispatcher design of FLIP-6 before I make further changes. Also we must decide whether the Mesos work should take FLIP-6 as a dependency, and/or strive to land a useable subset into master.
Here's a summary of the code I presented, as reference for ongoing FLIP-6 design discussion. A fresh PR for convenience: https://github.com/EronWright/flink/pull/1/files 1. MesosDispatcher (Backend). The 'backend' acts as a Mesos framework, to launch an AppMaster as a Mesos task. For each session, the backend accepts "session parameters" which define the libraries, configuration, resource profile and other parameters needed to launch the AppMaster. The backend persists the information for recovery purposes. Leader election is also used. Not included is a dispatcher 'frontend' - a formalized API surface or REST server. 2. SessionParameters. Captures the information needed to launch a session based on an AppMaster. A session is a stateful execution environment for a program. The session parameters can also be understood as the historical inputs to yarn-session.sh, plus job inputs per FLIP-6. I acknowledge that the term 'session' is a working term. 3 . MesosDispatcherRunner. The runner for the 'remote dispatcher' scenario, which would be started by Marathon, expose a REST API with which to submit/manage jobs, and host the above dispatcher backend. 4. FlinkMesosSessionCli. This class mirrors the FlinkYarnSessionCli, which is used in both the 'flink run' scenario and 'yarn-session' scenario. I didn't fully implement the CustomCommandLine interface which yields a ClusterClient, because the dispatcher API must be fleshed out first. 5. SessionArtifactHelper. An attempt to consolidate logic related to session artifacts (i.e. ship files). 6. DispatcherClient. The client interface for the dispatcher. In concept there could be numerous implementations - a 'remote' impl which would make REST calls to a remote dispatcher, a 'local' impl which would host the dispatcher directly. Seen in this PR is only the latter but it might be throw-away code. 7. LaunchableMesosSession. This class generates the effective container environment at launch time. 8. ContaineredJobMasterParameters. Refactored from YARN code for sharing purposes. -Eron
