Hi Navneet, Your payload size is larger than the maximum allowed size of a message on the Producer/Broker, 1000000 is the current default value. Currently the check for max allowed message size is present on both the producer and the broker. So you can override the property "max.message.size" in ProducerConfig and KafkaConfig to the same increased size. Additionally, you need to make sure that "fetch.size" in ConsumerConfig is at least as much as the max message size on producer, because the messages produced by the producer should be small enough for any consumer that might read them.
Alternately, you can reduce the payload size of the message. Thanks, Swapnil On 4/18/13 11:17 PM, "navneet sharma" <navneetsharma0...@gmail.com> wrote: >Hi, > >I am getting an exception while working with a text file. > >kafka.common.MessageSizeTooLargeException: payload size of 1839314 larger >than 1000000 >at >kafka.message.ByteBufferMessageSet.verifyMessageSize(ByteBufferMessageSet. >scala:93) >at kafka.producer.SyncProducer.send(SyncProducer.scala:122) >at >kafka.producer.ProducerPool$$anonfun$send$1.apply$mcVI$sp(ProducerPool.sca >la:114) >at >kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:100) >at >kafka.producer.ProducerPool$$anonfun$send$1.apply(ProducerPool.scala:100) >at >scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala >:60) >at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) >at kafka.producer.ProducerPool.send(ProducerPool.scala:100) >at kafka.producer.Producer.zkSend(Producer.scala:137) >at kafka.producer.Producer.send(Producer.scala:99) >at kafka.javaapi.producer.Producer.send(Producer.scala:103) >at >com.symphony.bigdata.analytics.kafka.SamCustomProducer$2.run(SamCustomProd >ucer.java:114) >at >java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor. >java:886) >at >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java >:908) >at java.lang.Thread.run(Thread.java:662) > >Any pointer what might be the issue and how to fix it. I am using 0.7.2 >version > >Thanks, >Navneet Sharma
