Re: separation of JVMs for different applications

2016-12-10 Thread Manu Zhang
Created https://issues.apache.org/jira/browse/FLINK-5312. Thanks, Manu On Fri, Dec 9, 2016 at 7:17 PM Till Rohrmann wrote: > Hi Manu, > > afaik there is no JIRA for standalone v2.0 yet. So feel free to open an > JIRA for it. > > Just a small correction, FLIP-6 is not almost finished yet. But we

Re: separation of JVMs for different applications

2016-12-09 Thread Till Rohrmann
Hi Manu, afaik there is no JIRA for standalone v2.0 yet. So feel free to open an JIRA for it. Just a small correction, FLIP-6 is not almost finished yet. But we're working on it and are happy for every helping hand :-) Cheers, Till On Fri, Dec 9, 2016 at 2:27 AM, Manu Zhang wrote: > If there

Re: separation of JVMs for different applications

2016-12-08 Thread Manu Zhang
If there are not any existing jira for standalone v2.0, may I open a new one ? Thanks, Manu On Wed, Dec 7, 2016 at 12:39 PM Manu Zhang wrote: > Good to know that. > > Is it the "standalone setup v2.0" section ? The wiki page has no > Google-Doc-like change histories. > Any jiras opened for that

Re: separation of JVMs for different applications

2016-12-06 Thread Manu Zhang
Good to know that. Is it the "standalone setup v2.0" section ? The wiki page has no Google-Doc-like change histories. Any jiras opened for that ? Not sure that will be noticed given FLIP-6 is almost finished. Thanks, Manu On Tue, Dec 6, 2016 at 11:55 PM Stephan Ewen wrote: > Hi! > > We are cur

Re: separation of JVMs for different applications

2016-12-06 Thread Stephan Ewen
Hi! We are currently changing the resource and process model quite a bit: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 As part of that, I think it makes sense to introduce something like that. What you can do today is to set TaskManagers to use one slot only, and then

Re: separation of JVMs for different applications

2016-12-05 Thread Manu Zhang
Thanks Stephan, They don't use YARN now but I think they will consider it. Do you think it would be beneficial to provide such an option as "separate-jvm" in stand-alone mode for streaming processor and long running services ? Or do you think it would introduce too much complexity ? Manu On Tue

Re: separation of JVMs for different applications

2016-12-05 Thread Stephan Ewen
Hi! Are your customers using YARN? In that case, the default configuration will start a new YARN application per Flink job, no JVMs are shared between jobs. By default, even each slot has its own JVM. Greetings, Stephan PS: I think the "spawning new JVMs" is what Till referred to when saying "sp

Re: separation of JVMs for different applications

2016-12-05 Thread Manu Zhang
> > The pro for the multi-tenant cluster mode is that you can share data > between jobs and you don't have to spin up a new cluster for each job. I don't think we have to spin up a new cluster for each job if every job gets its own JVMs. For examples, Storm will launch a new worker(JVM) for a new

Re: separation of JVMs for different applications

2016-12-05 Thread Till Rohrmann
The pro for the multi-tenant cluster mode is that you can share data between jobs and you don't have to spin up a new cluster for each job. This might be helpful for scenarios where you want to run many short-lived and light-weight jobs. But the important part is that you don't have to use this me

Re: separation of JVMs for different applications

2016-12-03 Thread Manu Zhang
Thanks Fabian and Till. We have customers who are interested in using Flink but very concerned about that "multiple jobs share the same set of TMs". I've just joined the community recently so I'm not sure whether there has been a discussion over the "multi-tenant cluster mode" before. The cons ar

Re: separation of JVMs for different applications

2016-12-02 Thread Till Rohrmann
Hi Manu, with Flip-6 we will be able to support stricter application isolation by starting for each job a dedicated JobManager which will execute its tasks on TM reserved solely for this job. But at the same time we will continue supporting the multi-tenant cluster mode where tasks belonging to mu

Re: separation of JVMs for different applications

2016-12-02 Thread Fabian Hueske
Hi Manu, As far as I know, there are not plans to change the stand-alone deployment. FLIP-6 is focusing on deployments via resource providers (YARN, Mesos, etc.) which allow to start Flink processes per job. Till (in CC) is more familiar with the FLIP-6 effort and might be able to add more detail