Hi,
I am coming across a use case where I may have to run more than100 parallel
jobs(which may have different processing needs) on a Flink cluster.
My flink cluster, currently, has 1 Job Manager and 4/5 Task Managers depending
on the processing needed is running on a Kubernetes cluster with 3 worker
nodes.
I have few questions around this before I start looking at how I can execute
this use case on such cluster configuration :
1) Is there any limitation w.r.t Job Manager while running 100 parallel jobs
on a similar cluster? 2) Is there any limitation on the number of jobs that can
be run on a cluster with 1 Job Manager? 3) Are there any recommendations around
having hundreds of jobs on a single cluster?4) Would I need multiple clusters
for such a use case or are there any tuning parameters which can help for eg.
Akka tuning parameters?
Thanks in advance,
Chirag
