i m trying to read more than 1mb msg from kafka using flume
and i m getting fetch size error.
where to define in flume config for fetch.message.max.bytes.
error:
kafka.common.MessageSizeTooLargeException: Found a message larger than the
maximum fetch size of this consumer on topic test partition 2 at fetch
offset 0. Increase the fetch size, or decrease the maximum message size the
broker will allow.
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:91)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
at
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
at
org.apache.flume.channel.kafka.KafkaChannel$KafkaTransaction.doTake(KafkaChannel.java:306)
at
org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
at
org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
at
org.apache.flume.sink.hdfs.HDFSEventSink1.process(HDFSEventSink1.java:374)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
