Hi,
Sharing new findings.
The issue I have mentioned above seems to be happening only with the latest
version of EMR(emr-5.20.0, hadoop: Amazon 2.8.5, Flink: 1.6.2) and it is
reproducible with our setup every time. I have verified the same setup
working and scaling without any issues on an older
Hi,
I am using Amazon EMR(emr-5.20.0, hadoop: Amazon 2.8.5, Flink: 1.6.2) to run
Flink Cluster on YARN. My setup consists of m4.large instances for 1 master
and 3 core nodes. I have Flink Cluster running on YARN with the command:
flink-yarn-session -tm 5120 -s 3 -jm 1024. This setup should ideall
Hi,
I'm using Flink 1.5 with RocksDB State Backend for Checkpointing. According
to the docs, Checkpoints do not support Job rescaling. However, when i tried
to restart a job from an externalised checkpoint with a different
parallelism, the job resumes correctly without any issues.
I would like to
Hi Till,
What I was ideally looking for was to have a completely managed service for
Flink via AWS EMR in which YARN Cluster would be completely dedicated to
only one Flink Session and as the EMR scales in and out, EMR/YARN would
add/remove TMs accordingly. I could then get the value of total task
Thanks for the clarification, Dawid and Till.
@Till We have a few streaming jobs that need to be running all the time and
we plan on using the modify tool to update parallelism of jobs as we scale
the cluster in and out and knowing total slots value is crucial to this
workflow.
As Dawid pointed o
Hi,
I am using Amazon EMR to run Flink Cluster on YARN. My setup consists of
m4.large instances for 1 master and 2 core nodes. I have started the Flink
Cluster on YARN with the command: flink-yarn-session -n 2 -d -tm 4096 -s 4.
Flink Job Manager and Application Manager starts but there are no Tas
