Hi all,
Zhu Zhu and I propose to introduce a new job scheduler to Flink: adaptive batch job scheduler. The new scheduler can automatically decide parallelisms of job vertices for batch jobs, according to the size of data volume each vertex needs to process. Major benefits of this scheduler includes: 1. Batch job users can be relieved from parallelism tuning 2. Automatically tuned parallelisms can be vertex level and can better fit consumed datasets which have a varying volume size every day 1. Vertices from SQL batch jobs can be assigned with different parallelisms which are automatically tuned 2. It can be the first step towards enabling auto-rebalancing workloads of tasks You can find more details in the FLIP-187[1]. Looking forward to your feedback. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Job+Scheduler Best, Lijie
