Re: Auto/Dynamic scaling in Flink

2018-11-15 Thread 罗齐
Hi Nauroz, If you’re using Flink 1.5 on Yarn, it supports dynamic task manager allocation by default [1]. After skimming the code, it seems to me that in general if requested parallelism is larger than available task slots, new task managers will be requested via ResourceManager (please correct

Re: Auto/Dynamic scaling in Flink

2018-11-13 Thread Tzu-Li Chen
Hi, Yet Flink does not support auto-scaling. However, there is an umbrella JIRA issue[1] to cover the discussion about it. And I think the design doc(draft) attached[2] could help. Best, tison. [1] https://issues.apache.org/jira/browse/FLINK-10407 [2] https://docs.google.com/document/d/1XKDXnrp8

Re: Auto/Dynamic scaling in Flink

2018-11-13 Thread Tzu-Li (Gordon) Tai
Hi, Flink does not support auto-scaling, yet. Rescaling operations currently are always manual, i.e take a savepoint of the Flink job, and when restoring from the savepoint, define a new parallelism for the job. As for the metrics to be used for auto-scaling, I can imagine that it would be possibl

Auto/Dynamic scaling in Flink

2018-11-12 Thread Nauroz Khan Nausherwani
Dear Flink Contributors and users, I am a PhD student and I was interested to know, using which matrices, and when does Flink performs scaling-in or scaling out of resources? I did search the flink's website where I could only find information about how dynamic scaling is performed in stateless