Hi Xintong, Thx for your reply. Increasing network memory buffers (fraction, min, max) seems to increase tasks slightly.
Streaming job Standalone Vijay On Fri, May 22, 2020 at 2:49 AM Xintong Song <tonysong...@gmail.com> wrote: > Hi Vijay, > > I don't think your problem is related to number of opening files. The > parallelism of your job is decided before actually tries to open the files. > And if the OS limit for opening files is reached, you should see a job > execution failure, instead of a success execution with a lower parallelism. > > Could you share some more information about your use case? > > - What kind of job are your executing? Is it a streaming or batch > processing job? > - Which Flink deployment do you use? Standalone? Yarn? > - It would be helpful if you can share the Flink logs. > > > Thank you~ > > Xintong Song > > > > On Wed, May 20, 2020 at 11:50 PM Vijay Balakrishnan <bvija...@gmail.com> > wrote: > >> Hi, >> I have increased the number of slots available but the Job is not using >> all the slots but runs into this approximate 18000 Tasks limit. Looking >> into the source code, it seems to be opening file - >> https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/FileOutputFormat.java#L203 >> So, do I have to tune the ulimit or something similar at the Ubuntu O/S >> level to increase number of tasks available ? What I am confused about is >> the ulimit is per machine but the ExecutionGraph is across many machines ? >> Please pardon my ignorance here. Does number of tasks equate to number of >> open files. I am using 15 slots per TaskManager on AWS m5.4xlarge which has >> 16 vCPUs. >> >> TIA. >> >> On Tue, May 19, 2020 at 3:22 PM Vijay Balakrishnan <bvija...@gmail.com> >> wrote: >> >>> Hi, >>> >>> Flink Dashboard UI seems to show tasks having a hard limit for Tasks >>> column around 18000 on a Ubuntu Linux box. >>> I kept increasing the number of slots per task manager to 15 and number >>> of slots increased to 705 but the slots to tasks >>> stayed at around 18000. Below 18000 tasks, the Flink Job is able to >>> start up. >>> Even though I increased the number of slots, it still works when 312 >>> slots are being used. >>> >>> taskmanager.numberOfTaskSlots: 15 >>> >>> What knob can I tune to increase the number of Tasks ? >>> >>> Pls find attached the Flink Dashboard UI. >>> >>> TIA, >>> >>>
