Thank you Gary. That was helpful.
Thanks,
Jins George
On 2/17/19 10:03 AM, Gary Yao wrote:
Hi Jins George,
Every TM brings additional overhead, e.g., more heartbeat messages. However, a
cluster with 28 TMs would not be considered big as there are users that are
running Flink applications on tho
Hi Jins George,
Every TM brings additional overhead, e.g., more heartbeat messages.
However, a
cluster with 28 TMs would not be considered big as there are users that are
running Flink applications on thousands of cores [1][2].
Best,
Gary
[1]
https://flink.apache.org/flink-architecture.html#run-
Thanks Gary. Understood the behavior.
I am leaning towards running 7 TM on each machine(8 core), I have 4 nodes, that
will end up 28 taskmanagers and 1 job manager. I was wondering if this can
bring additional burden on jobmanager? Is it recommended?
Thanks,
Jins George
On 2/14/19 8:49 AM, Ga
Hi Jins George,
This has been asked before [1]. The bottom line is that you currently cannot
pre-allocate TMs and distribute your tasks evenly. You might be able to
achieve a better distribution across hosts by configuring fewer slots in
your
TMs.
Best,
Gary
[1]
http://apache-flink-user-mailing-
Hi,
I'm forwarding this question to Gary (CC'ed), who most likely would have an
answer for your question here.
Cheers,
Gordon
On Wed, Feb 13, 2019 at 8:33 AM Jins George wrote:
> Hello community,
>
> I am trying to upgrade a Flink Yarn session cluster running BEAM
> pipelines from version 1
Hello community,
I am trying to upgrade a Flink Yarn session cluster running BEAM pipelines
from version 1.2.0 to 1.6.3.
Here is my session start command: yarn-session.sh -d -n 4 -jm 1024 -tm 3072 -s
7
Because of the dynamic resource allocation, no taskmanager gets created
initially. Now
