Hi, > I wonder if I can tune the number of task managers? Is there a corresponding > config?
With K8S/Yarn resource provider, the task managers are allocated on demand. So, the number of them are depends on the max parallelism and the slot sharing group topology of your job. In standalone mode, you need to config the "conf/workers" in your flink distribution to decide the number of task managers[3]. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/ [2] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/ [3] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/#starting-and-stopping-a-cluster Best, Yangze Guo Best, Yangze Guo On Fri, May 7, 2021 at 7:34 PM Tamir Sagi <[email protected]> wrote: > > Hey > > num of TMs = parallelism / num of slots > > parallelism.default is another config you should consider. > > Read also > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/execution/parallel/ > > > ________________________________ > From: Yik San Chan <[email protected]> > Sent: Friday, May 7, 2021 1:56 PM > To: user <[email protected]> > Subject: How to increase the number of task managers? > > > EXTERNAL EMAIL > > > > Hi community, > > According to the > [docs](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/): > > > taskmanager.numberOfTaskSlots: The number of slots that a TaskManager > > offers (default: 1). Each slot can take one task or pipeline. Having > > multiple slots in a TaskManager can help amortize certain constant > > overheads (of the JVM, application libraries, or network connections) > > across parallel tasks or pipelines. See the Task Slots and Resources > > concepts section for details. > > > Running more smaller TaskManagers with one slot each is a good starting > > point and leads to the best isolation between tasks. Dedicating the same > > resources to fewer larger TaskManagers with more slots can help to increase > > resource utilization, at the cost of weaker isolation between the tasks > > (more tasks share the same JVM). > > We're able to tune slot count by setting taskmanager.numberOfTaskSlots, that > may help parallelize my task. > > I wonder if I can tune the number of task managers? Is there a corresponding > config? > > Best, > Yik San > > > Confidentiality: This communication and any attachments are intended for the > above-named persons only and may be confidential and/or legally privileged. > Any opinions expressed in this communication are not necessarily those of > NICE Actimize. If this communication has come to you in error you must take > no action based on it, nor must you copy or show it to anyone; please > delete/destroy and inform the sender by e-mail immediately. > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. > Viruses: Although we have taken steps toward ensuring that this e-mail and > attachments are free from any virus, we advise that in keeping with good > computing practice the recipient should ensure they are actually virus free.
