Have you tried getting the memory usage output using tool like jmap and
seeing what's consuming the memory? Also, what are you heap sizes for
the process?
-Jaikiran
On Tuesday 19 April 2016 02:31 AM, McKoy, Nick wrote:
To follow up with my last email, I have been looking into
socket.receive.buffer.byte as well as socket.send.buffer.bytes. Would it help
to increase the buffer for OOM issue?
All help is appreciated!
Thanks!
-nick
From: "McKoy, Nick"
<nicholas.mc...@washpost.com<mailto:nicholas.mc...@washpost.com>>
Date: Monday, April 18, 2016 at 3:41 PM
To: "users@kafka.apache.org<mailto:users@kafka.apache.org>"
<users@kafka.apache.org<mailto:users@kafka.apache.org>>
Subject: Out of memory - Java Heap space
Hey all,
I have a kafka cluster of 5 nodes that’s working really hard. CPU is around 40%
idle daily.
I looked at the file descriptor note on this documentation page
http://docs.confluent.io/1.0/kafka/deployment.html#file-descriptors-and-mmap
and decided to give it a shot on one instance in the cluster just to see how it
performed. I increased this number to 1048576.
I kept getting this error from the kafka logs:
ERROR [ReplicaFetcherThread--1-6], Error due to
(kafka.server.ReplicaFetcherThread) java.lang.OutOfMemoryError: Java heap space
I increased heap to see if that would help and I kept seeing these errors.
Could the file descriptor change have something related to this?
—
Nicholas McKoy
Engineering – Big Data and Personalization
Washington Post Media
One Franklin Square, Washington, DC 20001
Email: nicholas.mc...@washpost.com<mailto:nicholas.mc...@washpost.com>
