Jun, The peak sending rate is about 200/s(there are 3 SAS disks for 3 partitions). We use sync producer for we can not tolerate message discarding. The truth is our messages' producing rate is increasing these days and the kafka had been running perfectly for a long time before. I'll use a bigger heap size and monitor for a period. On such a producing rate and message size, is there any advices for config files(e.g socket buffer) you can give me? Thanks.
2012/12/27 Jun Rao <jun...@gmail.com> > 400KB messages are relatively large. How many messages are you sending per > sec? How big is your JVM heap? You may need a bigger heap size. > > Thanks, > > Jun > > On Wed, Dec 26, 2012 at 5:41 PM, xingcan <xingc...@gmail.com> wrote: > > > Hi, > > > > I run 'kafka-0.7.2' on a single node and today I got some error logs > below. > > > > ... > > [2012-12-26 08:51:18,496] INFO End registering broker topic > > /brokers/topics/jnits/1 (kafka.server.KafkaZooKeeper) > > 161557 [2012-12-26 08:51:19,103] INFO Closing socket connection to / > > 172.18.116.52. (kafka.network.Processor) > > 161558 [2012-12-26 08:51:18,416] ERROR OOME with size 313705 > > (kafka.network.BoundedByteBufferReceive) > > 161559 java.lang.OutOfMemoryError: Java heap space > > 161560 [2012-12-26 08:51:18,341] ERROR OOME with size 317477 > > (kafka.network.BoundedByteBufferReceive) > > 161561 java.lang.OutOfMemoryError: Java heap space > > 161562 [2012-12-26 08:51:19,178] ERROR Closing socket for > /172.18.0.34because > > of error (kafka.network.Processor) > > 161563 java.lang.OutOfMemoryError: Java heap space > > 161564 [2012-12-26 08:51:19,178] ERROR Closing socket for > > /172.18.116.46 because > > of error (kafka.network.Processor) > > 161565 java.lang.OutOfMemoryError: Java heap space > > 161566 [2012-12-26 08:51:19,375] ERROR OOME with size 269336 > > (kafka.network.BoundedByteBufferReceive) > > 161567 java.lang.OutOfMemoryError: Java heap space > > 161568 [2012-12-26 08:51:19,375] ERROR Closing socket for > > /172.18.113.38 because > > of error (kafka.network.Processor) > > 161569 java.lang.OutOfMemoryError: Java heap space > > ... > > > > The size of each message is about 400KB. Maybe the messages are produced > > too fast or the disk can not be written at such a rate(about > 33MB/s/disk)? > > Can anyone help me explain it? Thanks a lot. > > -- > > *Xingcan* > > > -- *Xingcan*
