Maybe you are exhausting your sockets, not file handles for some reason?
________________________________________
From: Kessiler Rodrigues [kessi...@callinize.com]
Sent: 31 July 2016 22:14
To: users@kafka.apache.org
Subject: Re: Too Many Open Files
I’m still experiencing this issue…
Here are the kafka logs.
[2016-07-31 20:10:35,658] ERROR Error while accepting connection
(kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:323)
at kafka.network.Acceptor.run(SocketServer.scala:268)
at java.lang.Thread.run(Thread.java:745)
[2016-07-31 20:10:35,658] ERROR Error while accepting connection
(kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:323)
at kafka.network.Acceptor.run(SocketServer.scala:268)
at java.lang.Thread.run(Thread.java:745)
[2016-07-31 20:10:35,658] ERROR Error while accepting connection
(kafka.network.Acceptor)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at kafka.network.Acceptor.accept(SocketServer.scala:323)
at kafka.network.Acceptor.run(SocketServer.scala:268)
at java.lang.Thread.run(Thread.java:745)
My ulimit is 1 million, how is that possible?
Can someone help with this?
> On Jul 30, 2016, at 5:05 AM, Kessiler Rodrigues <kessi...@callinize.com>
> wrote:
>
> I have changed it a bit.
>
> I have 10 brokers and 20k topics with 1 partition each.
>
> I looked at the kaka’s logs dir and I only have 3318 files.
>
> I’m doing some tests to see how many topics/partitions I can have, but it is
> throwing too many files once it hits 15k topics..
>
> Any thoughts?
>
>
>
>> On Jul 29, 2016, at 10:33 PM, Gwen Shapira <g...@confluent.io> wrote:
>>
>> woah, it looks like you have 15,000 replicas per broker?
>>
>> You can go into the directory you configured for kafka's log.dir and
>> see how many files you have there. Depending on your segment size and
>> retention policy, you could have hundreds of files per partition
>> there...
>>
>> Make sure you have at least that many file handles and then also add
>> handles for the client connections.
>>
>> 1 million file handles sound like a lot, but you are running lots of
>> partitions per broker...
>>
>> We normally don't see more than maybe 4000 per broker and most
>> clusters have a lot fewer, so consider adding brokers and spreading
>> partitions around a bit.
>>
>> Gwen
>>
>> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
>> <kessi...@callinize.com> wrote:
>>> Hi guys,
>>>
>>> I have been experiencing some issues on kafka, where its throwing too many
>>> open files.
>>>
>>> I have around of 6k topics and 5 partitions each.
>>>
>>> My cluster was made with 6 brokers. All of them are running Ubuntu 16 and
>>> the file limits settings are:
>>>
>>> `cat /proc/sys/fs/file-max`
>>> 2000000
>>>
>>> `ulimit -n`
>>> 1000000
>>>
>>> Anyone has experienced it before?
>
