Kafka ERROR after upgrade to Kafka 1.0.0 version : java.lang.OutOfMemoryError: Java heap space

[Vinay Kumar]

Hi,

I upgraded Kafka from 0.10.2.1 to 1.0.0 version. And only from then, I'm
seeing the kafka service going down because of below issue:

ERROR [KafkaApi-1] Error when handling request ; (kafka.server.KafkaApis)
{replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic=
xxxx,partitions=[{partition=1,fetch_offset=847134777,max_bytes=1048576}]}]}
java.lang.OutOfMemoryError: Java heap space

Even after increasing the HEAP size (KAFKA_HEAP_OPTS) to 5G, still the same
error is coming (below is the error log)

Can you please help on this issue. The no. of partitions used is 3, and the
replication factor for the topic is 2.



[2017-11-17 10:53:32,178] ERROR [KafkaApi-0] Error when handling request
{replica_id=-1,max_wait_time=500,min_bytes=1,topics=[{topic=
xxxx,partitions=[{partition=2,fetch_offset=776723939,max_bytes=1048576}]}]}
(kafka.server.KafkaApis)

java.lang.OutOfMemoryError: Java heap space

        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)

        at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)

        at org.apache.kafka.common.record.AbstractRecords.downConvert(
AbstractRecords.java:101)

        at org.apache.kafka.common.record.FileRecords.downConvert(
FileRecords.java:253)

        at kafka.server.KafkaApis.$anonfun$handleFetchRequest$4(KafkaAp
is.scala:520)

        at kafka.server.KafkaApis.$anonfun$handleFetchRequest$4$adapted
(KafkaApis.scala:518)

        at kafka.server.KafkaApis$$Lambda$668/2135785513.apply(Unknown
Source)

        at scala.Option.map(Option.scala:146)

        at kafka.server.KafkaApis.$anonfun$handleFetchRequest$3(KafkaAp
is.scala:518)

        at kafka.server.KafkaApis.$anonfun$handleFetchRequest$3$adapted
(KafkaApis.scala:508)

        at kafka.server.KafkaApis$$Lambda$667/987137379.apply(Unknown
Source)

        at scala.Option.flatMap(Option.scala:171)

        at kafka.server.KafkaApis.convertedPartitionData$1(KafkaApis.
scala:508)

        at kafka.server.KafkaApis.$anonfun$handleFetchRequest$12(KafkaA
pis.scala:556)

        at kafka.server.KafkaApis$$Lambda$664/79178972.apply(Unknown Source)

        at scala.collection.Iterator.foreach(Iterator.scala:929)

        at scala.collection.Iterator.foreach$(Iterator.scala:929)

        at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)

        at scala.collection.IterableLike.foreach(IterableLike.scala:71)

        at scala.collection.IterableLike.foreach$(IterableLike.scala:70)

        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)

        at kafka.server.KafkaApis.createResponse$2(KafkaApis.scala:555)

        at kafka.server.KafkaApis.$anonfun$handleFetchRequest$14(KafkaA
pis.scala:569)

        at kafka.server.KafkaApis.$anonfun$handleFetchRequest$14$adapte
d(KafkaApis.scala:569)

        at kafka.server.KafkaApis$$Lambda$663/1244681576.apply(Unknown
Source)

        at kafka.server.KafkaApis.$anonfun$sendResponseMaybeThrottle$1(
KafkaApis.scala:2034)

        at kafka.server.KafkaApis$$Lambda$480/119699941.apply$mcVI$sp(Unknown
Source)

        at kafka.server.ClientRequestQuotaManager.maybeRecordAndThrottl
e(ClientRequestQuotaManager.scala:52)

        at kafka.server.KafkaApis.sendResponseMaybeThrottle(KafkaApis.
scala:2034)

        at kafka.server.KafkaApis.fetchResponseCallback$1(KafkaApis.
scala:569)

        at kafka.server.KafkaApis.$anonfun$handleFetchRequest$15(KafkaA
pis.scala:588)

        at kafka.server.KafkaApis$$Lambda$662/1703788464.apply$mcVI$sp(Unknown
Source)


Thanks,

Vinay Kumar