Hey all, I'm using bucketing sink with a bucketer that creates partition per customer per day. I sink the files to s3. it suppose to work on around 500 files at the same time (according to my partitioning).
I have a critical problem of 'Too many open files'. I've upload two taskmanagers, each with 16 slots. I've checked how many open files (or file descriptors) exist with 'lsof | wc -l' and it had reached over a million files on each taskmanager! after that, I'd decreased the num of taskSlots to 8 (4 in each taskmanager), and the concurrency dropped. checking 'lsof | wc -l' gave around 250k file on each machine. I also checked how many actual files exist in my tmp dir (it works on the files there before uploading them to s3) - around 3000. I think that each taskSlot works with several threads (maybe 16?), and each thread holds a fd for the actual file, and thats how the numbers get so high. Is that a know problem? is there anything I can do? by now, I filter just 10 customers and it works great, but I have to find a real solution so I can stream all the data. Maybe I can also work with a single task Slot per machine but I'm not sure this is a good idea. Thank you very much, Alon -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
