Hi! We are currently changing the resource and process model quite a bit: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 As part of that, I think it makes sense to introduce something like that.
What you can do today is to set TaskManagers to use one slot only, and then start multiple TaskManagers per machine. That makes sure that JVMs are never shared across machines. If you use the "start-cluster.sh" script from Flink, you can enter the same hostname multiple times in the workers file, and it will start multiple TaskManagers on a machine. Best, Stephan On Tue, Dec 6, 2016 at 3:51 AM, Manu Zhang <owenzhang1...@gmail.com> wrote: > Thanks Stephan, > > They don't use YARN now but I think they will consider it. Do you think > it would be beneficial to provide such an option as "separate-jvm" in > stand-alone mode for streaming processor and long running services ? Or do > you think it would introduce too much complexity ? > > Manu > > On Tue, Dec 6, 2016 at 1:04 AM Stephan Ewen <se...@apache.org> wrote: > >> Hi! >> >> Are your customers using YARN? In that case, the default configuration >> will start a new YARN application per Flink job, no JVMs are shared between >> jobs. By default, even each slot has its own JVM. >> >> Greetings, >> Stephan >> >> PS: I think the "spawning new JVMs" is what Till referred to when saying >> "spinning up a new cluster". Keep in mind that Flink is also a batch >> processor, and it handles sequences of short batch jobs (as issued for >> example by interactive shells) and it pre-allocates and manages a lot of >> memory for batch jobs. >> >> >> >> On Mon, Dec 5, 2016 at 3:48 PM, Manu Zhang <owenzhang1...@gmail.com> >> wrote: >> >> The pro for the multi-tenant cluster mode is that you can share data >> between jobs and you don't have to spin up a new cluster for each job. >> >> >> I don't think we have to spin up a new cluster for each job if every job >> gets its own JVMs. For examples, Storm will launch a new worker(JVM) for a >> new job when free slots are available. How can we share data between jobs >> and why ? >> >> >> >> On Mon, Dec 5, 2016 at 6:27 PM, Till Rohrmann <trohrm...@apache.org> >> wrote: >> >> The pro for the multi-tenant cluster mode is that you can share data >> between jobs and you don't have to spin up a new cluster for each job. This >> might be helpful for scenarios where you want to run many short-lived and >> light-weight jobs. >> >> But the important part is that you don't have to use this method. You can >> also start a new Flink cluster per job which will then execute the job >> isolated from any other jobs (given that you don't submit other jobs to >> this cluster). >> >> Cheers, >> Till >> >> On Sat, Dec 3, 2016 at 2:50 PM, Manu Zhang <owenzhang1...@gmail.com> >> wrote: >> >> Thanks Fabian and Till. >> >> We have customers who are interested in using Flink but very concerned >> about that "multiple jobs share the same set of TMs". I've just joined the >> community recently so I'm not sure whether there has been a discussion over >> the "multi-tenant cluster mode" before. >> >> The cons are one job/user's failure may crash another, which is >> unacceptable in a multi-tenant scenario. >> What are the pros ? Do the pros overweigh the cons ? >> >> Manu >> >> On Fri, Dec 2, 2016 at 7:06 PM Till Rohrmann <trohrm...@apache.org> >> wrote: >> >> Hi Manu, >> >> with Flip-6 we will be able to support stricter application isolation by >> starting for each job a dedicated JobManager which will execute its tasks >> on TM reserved solely for this job. But at the same time we will continue >> supporting the multi-tenant cluster mode where tasks belonging to multiple >> jobs share the same set of TMs and, thus, might share information between >> them. >> >> Cheers, >> Till >> >> On Fri, Dec 2, 2016 at 11:19 AM, Fabian Hueske <fhue...@gmail.com> wrote: >> >> Hi Manu, >> >> As far as I know, there are not plans to change the stand-alone >> deployment. >> FLIP-6 is focusing on deployments via resource providers (YARN, Mesos, >> etc.) which allow to start Flink processes per job. >> >> Till (in CC) is more familiar with the FLIP-6 effort and might be able to >> add more detail. >> >> Best, >> Fabian >> >> 2016-12-01 4:16 GMT+01:00 Manu Zhang <owenzhang1...@gmail.com>: >> >> Hi all, >> >> It seems tasks of different Flink applications can end up in the same JVM >> (TaskManager) in standalone mode. Isn't this fragile since errors in one >> application could crash another ? I checked FLIP-6 >> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077> >> but >> didn't found any mention of changing it in the future. >> >> Any thoughts or have I missed anything ? >> >> Thanks, >> Manu Zhang >> >> >> >> >> >> >>
