Hi Vijay, I don't think your problem is related to number of opening files. The parallelism of your job is decided before actually tries to open the files. And if the OS limit for opening files is reached, you should see a job execution failure, instead of a success execution with a lower parallelism.
Could you share some more information about your use case? - What kind of job are your executing? Is it a streaming or batch processing job? - Which Flink deployment do you use? Standalone? Yarn? - It would be helpful if you can share the Flink logs. Thank you~ Xintong Song On Wed, May 20, 2020 at 11:50 PM Vijay Balakrishnan <bvija...@gmail.com> wrote: > Hi, > I have increased the number of slots available but the Job is not using > all the slots but runs into this approximate 18000 Tasks limit. Looking > into the source code, it seems to be opening file - > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/FileOutputFormat.java#L203 > So, do I have to tune the ulimit or something similar at the Ubuntu O/S > level to increase number of tasks available ? What I am confused about is > the ulimit is per machine but the ExecutionGraph is across many machines ? > Please pardon my ignorance here. Does number of tasks equate to number of > open files. I am using 15 slots per TaskManager on AWS m5.4xlarge which has > 16 vCPUs. > > TIA. > > On Tue, May 19, 2020 at 3:22 PM Vijay Balakrishnan <bvija...@gmail.com> > wrote: > >> Hi, >> >> Flink Dashboard UI seems to show tasks having a hard limit for Tasks >> column around 18000 on a Ubuntu Linux box. >> I kept increasing the number of slots per task manager to 15 and number >> of slots increased to 705 but the slots to tasks >> stayed at around 18000. Below 18000 tasks, the Flink Job is able to start >> up. >> Even though I increased the number of slots, it still works when 312 >> slots are being used. >> >> taskmanager.numberOfTaskSlots: 15 >> >> What knob can I tune to increase the number of Tasks ? >> >> Pls find attached the Flink Dashboard UI. >> >> TIA, >> >>
