I agree with Steve. We had a similar problem where we set the ulimit to a certain value but it was getting overridden. It only worked when we set the ulimit after logging in as root. You might want to give that a try if you have not done so already
- Anirudh On Mon, Aug 1, 2016 at 1:19 PM, Steve Miller <st...@idrathernotsay.com> wrote: > Can you run lsof -p (pid) for whatever the pid is for your Kafka process? > > For the fd limits you've set, I don't think subtlety is required: if > there's a millionish lines in the output, the fd limit you set is where you > think it is, and if it's a lot lower than that, the limit isn't being > applied properly somehow (maybe you are running this under, say, > supervisord, and maybe its config is lowering the limit, or the limits for > root are as you say but the limits for the kafka user aren't being set > properly, that sort of thing). > > If you do have 1M lines in the output, at least this might give you a > place to start figuring out what's open and why. > > -Steve > > > On Jul 31, 2016, at 4:14 PM, Kessiler Rodrigues <kessi...@callinize.com> > wrote: > > > > I’m still experiencing this issue… > > > > Here are the kafka logs. > > > > [2016-07-31 20:10:35,658] ERROR Error while accepting connection > (kafka.network.Acceptor) > > java.io.IOException: Too many open files > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > > at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) > > at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) > > at kafka.network.Acceptor.accept(SocketServer.scala:323) > > at kafka.network.Acceptor.run(SocketServer.scala:268) > > at java.lang.Thread.run(Thread.java:745) > > [2016-07-31 20:10:35,658] ERROR Error while accepting connection > (kafka.network.Acceptor) > > java.io.IOException: Too many open files > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > > at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) > > at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) > > at kafka.network.Acceptor.accept(SocketServer.scala:323) > > at kafka.network.Acceptor.run(SocketServer.scala:268) > > at java.lang.Thread.run(Thread.java:745) > > [2016-07-31 20:10:35,658] ERROR Error while accepting connection > (kafka.network.Acceptor) > > java.io.IOException: Too many open files > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > > at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) > > at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) > > at kafka.network.Acceptor.accept(SocketServer.scala:323) > > at kafka.network.Acceptor.run(SocketServer.scala:268) > > at java.lang.Thread.run(Thread.java:745) > > > > My ulimit is 1 million, how is that possible? > > > > Can someone help with this? > > > > > >> On Jul 30, 2016, at 5:05 AM, Kessiler Rodrigues <kessi...@callinize.com> > wrote: > >> > >> I have changed it a bit. > >> > >> I have 10 brokers and 20k topics with 1 partition each. > >> > >> I looked at the kaka’s logs dir and I only have 3318 files. > >> > >> I’m doing some tests to see how many topics/partitions I can have, but > it is throwing too many files once it hits 15k topics.. > >> > >> Any thoughts? > >> > >> > >> > >>> On Jul 29, 2016, at 10:33 PM, Gwen Shapira <g...@confluent.io> wrote: > >>> > >>> woah, it looks like you have 15,000 replicas per broker? > >>> > >>> You can go into the directory you configured for kafka's log.dir and > >>> see how many files you have there. Depending on your segment size and > >>> retention policy, you could have hundreds of files per partition > >>> there... > >>> > >>> Make sure you have at least that many file handles and then also add > >>> handles for the client connections. > >>> > >>> 1 million file handles sound like a lot, but you are running lots of > >>> partitions per broker... > >>> > >>> We normally don't see more than maybe 4000 per broker and most > >>> clusters have a lot fewer, so consider adding brokers and spreading > >>> partitions around a bit. > >>> > >>> Gwen > >>> > >>> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues > >>> <kessi...@callinize.com> wrote: > >>>> Hi guys, > >>>> > >>>> I have been experiencing some issues on kafka, where its throwing too > many open files. > >>>> > >>>> I have around of 6k topics and 5 partitions each. > >>>> > >>>> My cluster was made with 6 brokers. All of them are running Ubuntu 16 > and the file limits settings are: > >>>> > >>>> `cat /proc/sys/fs/file-max` > >>>> 2000000 > >>>> > >>>> `ulimit -n` > >>>> 1000000 > >>>> > >>>> Anyone has experienced it before? > > > >
