Hi Kafka Users,
We are using kafka 0.9.0.1 and frequently we have seen below exception
which causes the broker
to die.We even increased the MaxDirectMemory to 1G but still see this.
2017-02-16 00:55:57,750] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Unknown Source)
at java.nio.DirectByteBuffer.<init>(Unknown Source)
at java.nio.ByteBuffer.allocateDirect(Unknown Source)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at
org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:108)
at
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:97)
at
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
at
org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153)
at
org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134)
at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
at kafka.network.Processor.run(SocketServer.scala:413)
at java.lang.Thread.run(Unknown Source)
Any pointers to what other metrics/configuration we can check to
determine the root cause
of this problem?
We have 32 nodes with 2 TB each and we are ingesting approx 3TB per
day into kafka.(Also reading
the same amount of data from hadoop MR jobs)
queue replication is 3. Most other parameters are default. Kafka
broker process hasXmx=1G
and MaxDirectMemory=1G
Thanks,-Vinay
