What happens if you change the consumer to write to disk or to the console (if text)? Do you still have the issues? I'm wondering if it's your algorithms that are taking up so much memory. I would say before you start tweaking options randomly, start with the simplest case.
Casey ________________________________________ From: Jun Rao [jun...@gmail.com] Sent: Thursday, December 27, 2012 1:25 AM To: users@kafka.apache.org Cc: Kafka-users Subject: Re: A "Java heap space" question 400KB messages are relatively large. How many messages are you sending per sec? How big is your JVM heap? You may need a bigger heap size. Thanks, Jun On Wed, Dec 26, 2012 at 5:41 PM, xingcan <xingc...@gmail.com> wrote: > Hi, > > I run 'kafka-0.7.2' on a single node and today I got some error logs below. > > ... > [2012-12-26 08:51:18,496] INFO End registering broker topic > /brokers/topics/jnits/1 (kafka.server.KafkaZooKeeper) > 161557 [2012-12-26 08:51:19,103] INFO Closing socket connection to / > 172.18.116.52. (kafka.network.Processor) > 161558 [2012-12-26 08:51:18,416] ERROR OOME with size 313705 > (kafka.network.BoundedByteBufferReceive) > 161559 java.lang.OutOfMemoryError: Java heap space > 161560 [2012-12-26 08:51:18,341] ERROR OOME with size 317477 > (kafka.network.BoundedByteBufferReceive) > 161561 java.lang.OutOfMemoryError: Java heap space > 161562 [2012-12-26 08:51:19,178] ERROR Closing socket for /172.18.0.34because > of error (kafka.network.Processor) > 161563 java.lang.OutOfMemoryError: Java heap space > 161564 [2012-12-26 08:51:19,178] ERROR Closing socket for > /172.18.116.46 because > of error (kafka.network.Processor) > 161565 java.lang.OutOfMemoryError: Java heap space > 161566 [2012-12-26 08:51:19,375] ERROR OOME with size 269336 > (kafka.network.BoundedByteBufferReceive) > 161567 java.lang.OutOfMemoryError: Java heap space > 161568 [2012-12-26 08:51:19,375] ERROR Closing socket for > /172.18.113.38 because > of error (kafka.network.Processor) > 161569 java.lang.OutOfMemoryError: Java heap space > ... > > The size of each message is about 400KB. Maybe the messages are produced > too fast or the disk can not be written at such a rate(about 33MB/s/disk)? > Can anyone help me explain it? Thanks a lot. > -- > *Xingcan* >
