Thanks for the replies! I was rather hoping not to have to implement a side channel solution. :/
If we have to do this, we may use an HBase table with a TTL the same as our topic so the large objects are "gc'ed"... thoughts? On Tue, Oct 6, 2015 at 8:45 AM, Gwen Shapira <g...@confluent.io> wrote: > Storing large blobs in S3 or HDFS and placing URIs in Kafka is the most > common solution I've seen in use. > > On Tue, Oct 6, 2015 at 8:32 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > The best practice I think is to just put large objects in a blob store > > and have messages embed references to those blobs. Interestingly we > > ended up having to implement large-message-support at LinkedIn but for > > various reasons were forced to put messages inline (i.e., against the > > above recommendation). So we ended up having to break up large > > messages into smaller chunks. This obviously adds considerable > > complexity to the consumer since the checkpointing can become pretty > > complicated. There are other nuances as well - we can probably do a > > short talk on this at an upcoming meetup. > > > > Joel > > > > > > On Mon, Oct 5, 2015 at 9:31 PM, Rahul Jain <rahul...@gmail.com> wrote: > > > In addition to the config changes mentioned in that post, you may also > > have > > > to change producer config if you are using the new producer. > > > > > > Specifically, *max.request.size* and *request.timeout.ms > > > <http://request.timeout.ms>* have to be increased to allow the > producer > > to > > > send large messages. > > > > > > > > > On 6 Oct 2015 02:02, "James Cheng" <jch...@tivo.com> wrote: > > > > > >> Here’s an article that Gwen wrote earlier this year on handling large > > >> messages in Kafka. > > >> > > >> http://ingest.tips/2015/01/21/handling-large-messages-kafka/ > > >> > > >> -James > > >> > > >> > On Oct 5, 2015, at 11:20 AM, Pradeep Gollakota < > pradeep...@gmail.com> > > >> wrote: > > >> > > > >> > Fellow Kafkaers, > > >> > > > >> > We have a pretty heavyweight legacy event logging system for batch > > >> > processing. We're now sending the events into Kafka now for realtime > > >> > analytics. But we have some pretty large messages (> 40 MB). > > >> > > > >> > I'm wondering if any of you have use cases where you have to send > > large > > >> > messages to Kafka and how you're dealing with them. > > >> > > > >> > Thanks, > > >> > Pradeep > > >> > > >> > > >> ________________________________ > > >> > > >> This email and any attachments may contain confidential and privileged > > >> material for the sole use of the intended recipient. Any review, > > copying, > > >> or distribution of this email (or any attachments) by others is > > prohibited. > > >> If you are not the intended recipient, please contact the sender > > >> immediately and permanently delete this email and any attachments. No > > >> employee or agent of TiVo Inc. is authorized to conclude any binding > > >> agreement on behalf of TiVo Inc. by email. Binding agreements with > TiVo > > >> Inc. may only be made by a signed written agreement. > > >> > > >
