Hey num of TMs = parallelism / num of slots
parallelism.default is another config you should consider. Read also https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/execution/parallel/ [https://my-email-signature.link/signature.gif?u=1088647&e=151783011&v=daa41c995e0f03eb0cd358bef914ad1aaab37cbec8d5045736b62bf3f38f5497] ________________________________ From: Yik San Chan <evan.chanyik...@gmail.com> Sent: Friday, May 7, 2021 1:56 PM To: user <user@flink.apache.org> Subject: How to increase the number of task managers? EXTERNAL EMAIL Hi community, According to the [docs](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/): > taskmanager.numberOfTaskSlots: The number of slots that a TaskManager offers > (default: 1). Each slot can take one task or pipeline. Having multiple slots > in a TaskManager can help amortize certain constant overheads (of the JVM, > application libraries, or network connections) across parallel tasks or > pipelines. See the Task Slots and > Resources<https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/concepts/flink-architecture/#task-slots-and-resources> > concepts section for details. > Running more smaller TaskManagers with one slot each is a good starting point > and leads to the best isolation between tasks. Dedicating the same resources > to fewer larger TaskManagers with more slots can help to increase resource > utilization, at the cost of weaker isolation between the tasks (more tasks > share the same JVM). We're able to tune slot count by setting taskmanager.numberOfTaskSlots, that may help parallelize my task. I wonder if I can tune the number of task managers? Is there a corresponding config? Best, Yik San Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately. Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
