Hi Yangze, Thanks for the answer! That helps.
Best, Yik San On Sat, May 8, 2021 at 10:15 AM Yangze Guo <[email protected]> wrote: > Hi, > > > I wonder if I can tune the number of task managers? Is there a > corresponding config? > > With K8S/Yarn resource provider, the task managers are allocated on > demand. So, the number of them are depends on the max parallelism and > the slot sharing group topology of your job. > In standalone mode, you need to config the "conf/workers" in your > flink distribution to decide the number of task managers[3]. > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/ > [2] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/ > [3] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/#starting-and-stopping-a-cluster > > Best, > Yangze Guo > > Best, > Yangze Guo > > > On Fri, May 7, 2021 at 7:34 PM Tamir Sagi <[email protected]> > wrote: > > > > Hey > > > > num of TMs = parallelism / num of slots > > > > parallelism.default is another config you should consider. > > > > Read also > > > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/execution/parallel/ > > > > > > ________________________________ > > From: Yik San Chan <[email protected]> > > Sent: Friday, May 7, 2021 1:56 PM > > To: user <[email protected]> > > Subject: How to increase the number of task managers? > > > > > > EXTERNAL EMAIL > > > > > > > > Hi community, > > > > According to the [docs]( > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/ > ): > > > > > taskmanager.numberOfTaskSlots: The number of slots that a TaskManager > offers (default: 1). Each slot can take one task or pipeline. Having > multiple slots in a TaskManager can help amortize certain constant > overheads (of the JVM, application libraries, or network connections) > across parallel tasks or pipelines. See the Task Slots and Resources > concepts section for details. > > > > > Running more smaller TaskManagers with one slot each is a good > starting point and leads to the best isolation between tasks. Dedicating > the same resources to fewer larger TaskManagers with more slots can help to > increase resource utilization, at the cost of weaker isolation between the > tasks (more tasks share the same JVM). > > > > We're able to tune slot count by setting taskmanager.numberOfTaskSlots, > that may help parallelize my task. > > > > I wonder if I can tune the number of task managers? Is there a > corresponding config? > > > > Best, > > Yik San > > > > > > Confidentiality: This communication and any attachments are intended for > the above-named persons only and may be confidential and/or legally > privileged. Any opinions expressed in this communication are not > necessarily those of NICE Actimize. If this communication has come to you > in error you must take no action based on it, nor must you copy or show it > to anyone; please delete/destroy and inform the sender by e-mail > immediately. > > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. > > Viruses: Although we have taken steps toward ensuring that this e-mail > and attachments are free from any virus, we advise that in keeping with > good computing practice the recipient should ensure they are actually virus > free. >
