Thanks Stephan, They don't use YARN now but I think they will consider it. Do you think it would be beneficial to provide such an option as "separate-jvm" in stand-alone mode for streaming processor and long running services ? Or do you think it would introduce too much complexity ?
Manu On Tue, Dec 6, 2016 at 1:04 AM Stephan Ewen <se...@apache.org> wrote: > Hi! > > Are your customers using YARN? In that case, the default configuration > will start a new YARN application per Flink job, no JVMs are shared between > jobs. By default, even each slot has its own JVM. > > Greetings, > Stephan > > PS: I think the "spawning new JVMs" is what Till referred to when saying > "spinning up a new cluster". Keep in mind that Flink is also a batch > processor, and it handles sequences of short batch jobs (as issued for > example by interactive shells) and it pre-allocates and manages a lot of > memory for batch jobs. > > > > On Mon, Dec 5, 2016 at 3:48 PM, Manu Zhang <owenzhang1...@gmail.com> > wrote: > > The pro for the multi-tenant cluster mode is that you can share data > between jobs and you don't have to spin up a new cluster for each job. > > > I don't think we have to spin up a new cluster for each job if every job > gets its own JVMs. For examples, Storm will launch a new worker(JVM) for a > new job when free slots are available. How can we share data between jobs > and why ? > > > > On Mon, Dec 5, 2016 at 6:27 PM, Till Rohrmann <trohrm...@apache.org> > wrote: > > The pro for the multi-tenant cluster mode is that you can share data > between jobs and you don't have to spin up a new cluster for each job. This > might be helpful for scenarios where you want to run many short-lived and > light-weight jobs. > > But the important part is that you don't have to use this method. You can > also start a new Flink cluster per job which will then execute the job > isolated from any other jobs (given that you don't submit other jobs to > this cluster). > > Cheers, > Till > > On Sat, Dec 3, 2016 at 2:50 PM, Manu Zhang <owenzhang1...@gmail.com> > wrote: > > Thanks Fabian and Till. > > We have customers who are interested in using Flink but very concerned > about that "multiple jobs share the same set of TMs". I've just joined the > community recently so I'm not sure whether there has been a discussion over > the "multi-tenant cluster mode" before. > > The cons are one job/user's failure may crash another, which is > unacceptable in a multi-tenant scenario. > What are the pros ? Do the pros overweigh the cons ? > > Manu > > On Fri, Dec 2, 2016 at 7:06 PM Till Rohrmann <trohrm...@apache.org> wrote: > > Hi Manu, > > with Flip-6 we will be able to support stricter application isolation by > starting for each job a dedicated JobManager which will execute its tasks > on TM reserved solely for this job. But at the same time we will continue > supporting the multi-tenant cluster mode where tasks belonging to multiple > jobs share the same set of TMs and, thus, might share information between > them. > > Cheers, > Till > > On Fri, Dec 2, 2016 at 11:19 AM, Fabian Hueske <fhue...@gmail.com> wrote: > > Hi Manu, > > As far as I know, there are not plans to change the stand-alone deployment. > FLIP-6 is focusing on deployments via resource providers (YARN, Mesos, > etc.) which allow to start Flink processes per job. > > Till (in CC) is more familiar with the FLIP-6 effort and might be able to > add more detail. > > Best, > Fabian > > 2016-12-01 4:16 GMT+01:00 Manu Zhang <owenzhang1...@gmail.com>: > > Hi all, > > It seems tasks of different Flink applications can end up in the same JVM > (TaskManager) in standalone mode. Isn't this fragile since errors in one > application could crash another ? I checked FLIP-6 > <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077> > but > didn't found any mention of changing it in the future. > > Any thoughts or have I missed anything ? > > Thanks, > Manu Zhang > > > > > > >
