IIRC, just add it to your flume configs, eg. for a source: tier1.sources.src1.kafka.fetch.message.max.bytes=
Thanks, --tim On Tue, Jan 12, 2016 at 7:25 AM, manish jaiswal <manishsr...@gmail.com> wrote: > i m trying to read more than 1mb msg from kafka using flume > and i m getting fetch size error. > > where to define in flume config for fetch.message.max.bytes. > > > > > error: > kafka.common.MessageSizeTooLargeException: Found a message larger than the > maximum fetch size of this consumer on topic test partition 2 at fetch > offset 0. Increase the fetch size, or decrease the maximum message size the > broker will allow. > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:91) > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33) > at > kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66) > at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58) > at > org.apache.flume.channel.kafka.KafkaChannel$KafkaTransaction.doTake(KafkaChannel.java:306) > at > org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113) > at > org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95) > at > org.apache.flume.sink.hdfs.HDFSEventSink1.process(HDFSEventSink1.java:374) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:745)
