In addition to the config changes mentioned in that post, you may also have to change producer config if you are using the new producer.
Specifically, *max.request.size* and *request.timeout.ms <http://request.timeout.ms>* have to be increased to allow the producer to send large messages. On 6 Oct 2015 02:02, "James Cheng" <jch...@tivo.com> wrote: > Here’s an article that Gwen wrote earlier this year on handling large > messages in Kafka. > > http://ingest.tips/2015/01/21/handling-large-messages-kafka/ > > -James > > > On Oct 5, 2015, at 11:20 AM, Pradeep Gollakota <pradeep...@gmail.com> > wrote: > > > > Fellow Kafkaers, > > > > We have a pretty heavyweight legacy event logging system for batch > > processing. We're now sending the events into Kafka now for realtime > > analytics. But we have some pretty large messages (> 40 MB). > > > > I'm wondering if any of you have use cases where you have to send large > > messages to Kafka and how you're dealing with them. > > > > Thanks, > > Pradeep > > > ________________________________ > > This email and any attachments may contain confidential and privileged > material for the sole use of the intended recipient. Any review, copying, > or distribution of this email (or any attachments) by others is prohibited. > If you are not the intended recipient, please contact the sender > immediately and permanently delete this email and any attachments. No > employee or agent of TiVo Inc. is authorized to conclude any binding > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo > Inc. may only be made by a signed written agreement. >
