> > The pro for the multi-tenant cluster mode is that you can share data > between jobs and you don't have to spin up a new cluster for each job.
I don't think we have to spin up a new cluster for each job if every job gets its own JVMs. For examples, Storm will launch a new worker(JVM) for a new job when free slots are available. How can we share data between jobs and why ? On Mon, Dec 5, 2016 at 6:27 PM, Till Rohrmann <trohrm...@apache.org> wrote: > The pro for the multi-tenant cluster mode is that you can share data > between jobs and you don't have to spin up a new cluster for each job. This > might be helpful for scenarios where you want to run many short-lived and > light-weight jobs. > > But the important part is that you don't have to use this method. You can > also start a new Flink cluster per job which will then execute the job > isolated from any other jobs (given that you don't submit other jobs to > this cluster). > > Cheers, > Till > > On Sat, Dec 3, 2016 at 2:50 PM, Manu Zhang <owenzhang1...@gmail.com> > wrote: > >> Thanks Fabian and Till. >> >> We have customers who are interested in using Flink but very concerned >> about that "multiple jobs share the same set of TMs". I've just joined the >> community recently so I'm not sure whether there has been a discussion over >> the "multi-tenant cluster mode" before. >> >> The cons are one job/user's failure may crash another, which is >> unacceptable in a multi-tenant scenario. >> What are the pros ? Do the pros overweigh the cons ? >> >> Manu >> >> On Fri, Dec 2, 2016 at 7:06 PM Till Rohrmann <trohrm...@apache.org> >> wrote: >> >>> Hi Manu, >>> >>> with Flip-6 we will be able to support stricter application isolation by >>> starting for each job a dedicated JobManager which will execute its tasks >>> on TM reserved solely for this job. But at the same time we will continue >>> supporting the multi-tenant cluster mode where tasks belonging to multiple >>> jobs share the same set of TMs and, thus, might share information between >>> them. >>> >>> Cheers, >>> Till >>> >>> On Fri, Dec 2, 2016 at 11:19 AM, Fabian Hueske <fhue...@gmail.com> >>> wrote: >>> >>> Hi Manu, >>> >>> As far as I know, there are not plans to change the stand-alone >>> deployment. >>> FLIP-6 is focusing on deployments via resource providers (YARN, Mesos, >>> etc.) which allow to start Flink processes per job. >>> >>> Till (in CC) is more familiar with the FLIP-6 effort and might be able >>> to add more detail. >>> >>> Best, >>> Fabian >>> >>> 2016-12-01 4:16 GMT+01:00 Manu Zhang <owenzhang1...@gmail.com>: >>> >>> Hi all, >>> >>> It seems tasks of different Flink applications can end up in the same >>> JVM (TaskManager) in standalone mode. Isn't this fragile since errors in >>> one application could crash another ? I checked FLIP-6 >>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077> >>> but >>> didn't found any mention of changing it in the future. >>> >>> Any thoughts or have I missed anything ? >>> >>> Thanks, >>> Manu Zhang >>> >>> >>> >>> >
