Becket did a good talk at the last Kafka meetup on how Linked In handle the large message problem.
http://www.slideshare.net/JiangjieQin/handle-large-messages-in-apache-kafka-58692297 <http://www.slideshare.net/JiangjieQin/handle-large-messages-in-apache-kafka-58692297> > On 14 Mar 2016, at 09:42, Jens Rantil <jens.ran...@tink.se> wrote: > > Just making it more explicit: AFAIK, all Kafka consumers I've seen loads the > incoming messages into memory. Unless you make it possible to stream it do > disk or something you need to make sure your consumers has the available > memory. > > Cheers, > Jens > > On Fri, Mar 4, 2016 at 6:07 PM Cees de Groot <c...@pagerduty.com > <mailto:c...@pagerduty.com>> wrote: > 1GB sounds like a tad steep, you may want to do some testing, as Kafka > needs to be told that such large messages can arrive and broker will then > pre-allocate buffers for that. Personally, I'd stop short of low megabytes, > anything bigger can be dropped off in e.g. S3 and then you just queue a > link for further processing. > > I'm not saying it's impossible, Kafka handles large messages better than > most other tools out there, but you do want to do a test setup to make sure > that it'll handle the sort of traffic you fling at it in any case. > > On Fri, Mar 4, 2016 at 4:26 AM, Mahesh Dharmasena <mahesh....@gmail.com > <mailto:mahesh....@gmail.com>> > wrote: > > > We have a client with several thousand stores which send and receive > > messages to main system that resides on the headquarters. > > > > A single Store sends and receive around 50 to 100 messages per day. > > > > Average Message size could be from 2KB to 1GB. > > > > Please let me know whether I can adapt Apache Kafka for the solution? > > > > > > - Mahesh. > > > > > > -- > > *Cees de Groot* > PRINCIPAL SOFTWARE ENGINEER > [image: PagerDuty logo] <http://pagerduty.com/ <http://pagerduty.com/>> > pagerduty.com <http://pagerduty.com/> > c...@pagerduty.com <mailto:c...@pagerduty.com> <m...@pagerduty.com > <mailto:m...@pagerduty.com>> > +1(416)435-4085 > > [image: Twitter] <http://twitter.com/pagerduty > <http://twitter.com/pagerduty>>[image: FaceBook] > <https://www.facebook.com/PagerDuty > <https://www.facebook.com/PagerDuty>>[image: Google+] > <https://plus.google.com/114339089137644062989 > <https://plus.google.com/114339089137644062989>>[image: LinkedIn] > <https://www.linkedin.com/company/pagerduty > <https://www.linkedin.com/company/pagerduty>>[image: Blog] > <https://blog.pagerduty.com/ <https://blog.pagerduty.com/>> > -- > Henrik Hedvall > Lead Designer > henrik.hedv...@tink.se <mailto:henrik.hedv...@tink.se> > +46 72 505 57 59 > > Tink AB > Wallingatan 5 > 111 60 Stockholm, Sweden > www.tink.se <http://www.tink.se/> > > >
