Hi Zhu and Abhishek, Thanks for your response and pointers. It's correct, the count of parallelism will be the number of slot used for a pipeline. And, the number (or count) of the parallelism is also used to generate number of sub-tasks for each operator. In my case, I have parallelism of 60, it generates 60 sub-tasks for each operator. And so it'll be too much for one slot execute at least 60 sub-tasks. I am wondering if there is a way we can set number of generated sub-tasks, different than number of parallelism?
Cam Mach Software Engineer E-mail: cammac...@gmail.com Tel: 206 972 2768 On Sun, Aug 11, 2019 at 10:37 PM Zhu Zhu <reed...@gmail.com> wrote: > Hi Cam, > This case is expected due to slot sharing. > A slot can be shared by one instance of different tasks. So the used slot > is count of your max parallelism of a task. > You can specify the shared group with slotSharingGroup(String > slotSharingGroup) on operators. > > Thanks, > Zhu Zhu > > Abhishek Jain <abhijai...@gmail.com> 于2019年8月12日周一 下午1:23写道： > >> What you'se seeing is likely operator chaining. This is the default >> behaviour of grouping sub tasks to avoid transer overhead (from one slot to >> another). You can disable chaining if you need to. Please refer task and >> operator chains >> <https://ci.apache.org/projects/flink/flink-docs-stable/concepts/runtime.html#tasks-and-operator-chains> >> . >> >> - Abhishek >> >> On Mon, 12 Aug 2019 at 09:56, Cam Mach <cammac...@gmail.com> wrote: >> >>> Hello Flink expert, >>> >>> I have a cluster with 10 Task Managers, configured with 6 task slot >>> each, and a pipeline that has 13 tasks/operators with parallelism of 5. But >>> when running the pipeline I observer that only 5 slots are being used, the >>> other 55 slots are available/free. It should use all of my slots, right? >>> since I have 13 (tasks) x 5 = 65 sub-tasks? What are the configuration that >>> I missed in order to leverage all of the available slots for my pipelines? >>> >>> Thanks, >>> Cam >>> >>> >>
