Re:Re: yarn-session vs cluster per job for streaming jobs

2019-07-18 Thread Haibo Sun
HI, Maxim As far as I understand, it's hard to draw a simple conclusion that who's faster. If the job is smaller (for example, the vertex number and the parallelism are very small), the session is usually faster than the per-job mode. I think the session has the advantage of sharing AM and TM,

Re: yarn-session vs cluster per job for streaming jobs

2019-07-17 Thread Maxim Parkachov
Hi Haibo, thanks for tip, I almost forgot about max-attempts. I understood implication of running with one AM. Maybe my question was incorrect, but what would be faster (with regards to downtime of each job): 1. In case of yarn-session: Parallel cancel all jobs with savepoints, restart yarn-sess

yarn-session vs cluster per job for streaming jobs

2019-07-17 Thread Maxim Parkachov
Hi, I'm looking for advice on how to run flink streaming jobs on Yarn cluster in production environment. I tried in testing environment both approaches with HA mode, namely yarn session + multiple jobs vs cluster per job, both seems to work for my cases, with slight preference of yarn session mode