Too many open files

[Govindarajan Srinivasaraghavan]

Hi,

We have a streaming job that runs on flink in docker and checkpointing
happens every 10 seconds. After several starts and cancellations we are
facing this issue with file handles.

The job reads data from kafka, processes it and writes it back to kafka and
we are using RocksDB state backend. For now we have increased the number
file handles to resolve the problem but would like to know if this is
expected or is it an issue. Thanks.

java.io.FileNotFoundException:
/tmp/flink-io-b3043cd6-50c8-446a-8c25-fade1b1862c0/cb317fc2578db72b3046468948fa00f2f17039b6104e72fb8c58938e5869cfbc.0.buffer
(Too many open files)

        at java.io.RandomAccessFile.open0(Native Method)

        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)

        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)

        at
org.apache.flink.streaming.runtime.io.BufferSpiller.createSpillingChannel(BufferSpiller.java:259)

        at
org.apache.flink.streaming.runtime.io.BufferSpiller.<init>(BufferSpiller.java:120)

        at
org.apache.flink.streaming.runtime.io.BarrierBuffer.<init>(BarrierBuffer.java:149)

        at
org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.<init>(StreamTwoInputProcessor.java:147)

        at
org.apache.flink.streaming.runtime.tasks.TwoInputStreamTask.init(TwoInputStreamTask.java:79)

        at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:235)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)

        at java.lang.Thread.run(Thread.java:748)

Regards,
Govind