@Fabian: Small misunderstanding :) I suggested to get rid of the multi-user mode in a standalone setup because Flink's support is not sufficient enough. Implementing proper support, as Stephan mentioned, is some work. It's already done in YARN.
IMO we can save a lot of time, lines of code, and brain cells if we simply switch to a "single-job-at-a-time" execution mode. Otherwise we have to worry about sharing of resources, scheduling policies, process isolation, user rights management, session management etc. On the other hand, we do have some of it in place. It just depends if we are ok with delegating the mutli-user case to YARN or other resource managers like Mesos. On Thu, Apr 30, 2015 at 10:08 AM, Stephan Ewen <se...@apache.org> wrote: > Most components are written multi-job aware. > > The only thing that is not in there right now is scheduling policies for > fair resource sharing. This is important in shared clusters. > > Since YARN implements all those things (various job queues with different > priorities/policies etc), I suggest to not try and re-build it in Flink and > simply declare a JobManager a "single-job-at-a-time" manager. You can still > run an interactive session with many jobs one after another. > > > On Wed, Apr 29, 2015 at 7:07 PM, Maximilian Michels <m...@apache.org> > wrote: > > > > > > > However, dropping it completely instead of improving it would make > Flink > > > setups on dedicated clusters quite useless, right? > > > > > > > Not really, because you could also use YARN on dedicated clusters for > > proper multi-user support. > > > > On Wed, Apr 29, 2015 at 5:51 PM, Fabian Hueske <fhue...@gmail.com> > wrote: > > > > > I agree that Flink's multi-user support is not very good at the moment. > > > However, dropping it completely instead of improving it would make > Flink > > > setups on dedicated clusters quite useless, right? > > > > > > > > > 2015-04-29 17:33 GMT+02:00 Maximilian Michels <m...@apache.org>: > > > > > > > Hi everyone, > > > > > > > > Currently Flink accepts jobs from multiple clients and executes them > > > > concurrently if the resource limits are not exceeded. However, the > > > > multi-user support is very poor. We don't support queuing of jobs and > > > > concurrent jobs have to share resources in a nice way. Otherwise, > jobs > > > will > > > > fail. > > > > > > > > Using YARN, we circumvent these problems because it provides a proper > > > user > > > > and session management. I'm wondering now, should we get rid of the > > > pseudo > > > > multi-user mode and just support one user per Flink cluster instance? > > > > > > > > Best, > > > > Max > > > > > > > > PS: > > > > This question came up when I was working on a pull request to support > > > > backtracking intermediate results. I need to hold a copy of the full > > > > previous execution graph to resume from old results. With multiple > > users, > > > > we have to build in some kind of session management to archive old > > > > execution graphs. Otherwise, they will consume too much memory in the > > job > > > > manager. > > > > > > > > > >
