Hi Till,
What I was ideally looking for was to have a completely managed service for
Flink via AWS EMR in which YARN Cluster would be completely dedicated to
only one Flink Session and as the EMR scales in and out, EMR/YARN would
add/remove TMs accordingly. I could then get the value of total task
With Flink 1.5.x and 1.6.x you can put `mode: legacy` into your
flink-conf.yaml and it will start the old mode. Then you have the old
behaviour.
What do you mean with total slots? The current number of total slots? With
resource elasticity this number can of course change because if you don't
have
Thanks for the clarification, Dawid and Till.
@Till We have a few streaming jobs that need to be running all the time and
we plan on using the modify tool to update parallelism of jobs as we scale
the cluster in and out and knowing total slots value is crucial to this
workflow.
As Dawid pointed o
Hi Suraj,
at the moment Flink's new mode does not support such a behaviour. There are
plans to set a min number of running TaskManagers which won't be released.
But no work has been done in this direction yet, afaik. If you want, then
you can help the community with this effort.
Cheers,
Till
On
Hi Suraj,
As far as I know this was changed with FLIP-6 to allow dynamic resource
allocation.
Till, cced might know if there is a switch to restore old behavior or
are there plans to support it.
Best,
Dawid
On 24/09/18 12:24, suraj7 wrote:
> Hi,
>
> I am using Amazon EMR to run Flink Cluster o
