Re: Kafka Applicability - Large Messages

9am the the-life-cycle-o9f-programming-languages 9 Sent from mobile, please excuse typos. Original Message From: Ben Stopford Sent: Monday, March 14, 2016 08:19 AM To: users@kafka.apache.org Subject: Re: Kafka Applicability - Large Messages Becket did a good talk at the last

Re: Kafka Applicability - Large Messages

On Mon, Mar 14, 2016 at 5:42 AM, Jens Rantil wrote: > Just making it more explicit: AFAIK, all Kafka consumers I've seen loads > the incoming messages into memory. Unless you make it possible to stream it > do disk or something you need to make sure your consumers has the available > memory. > >

Re: Kafka Applicability - Large Messages

Becket did a good talk at the last Kafka meetup on how Linked In handle the large message problem. http://www.slideshare.net/JiangjieQin/handle-large-messages-in-apache-kafka-58692297 > On 14 Mar 2016, at 0

Re: Kafka Applicability - Large Messages

Just making it more explicit: AFAIK, all Kafka consumers I've seen loads the incoming messages into memory. Unless you make it possible to stream it do disk or something you need to make sure your consumers has the available memory. Cheers, Jens On Fri, Mar 4, 2016 at 6:07 PM Cees de Groot wrote

Re: Kafka Applicability - Large Messages

1GB sounds like a tad steep, you may want to do some testing, as Kafka needs to be told that such large messages can arrive and broker will then pre-allocate buffers for that. Personally, I'd stop short of low megabytes, anything bigger can be dropped off in e.g. S3 and then you just queue a link f

Re: Kafka Applicability - Large Messages

I have used messages upto 20MB and while not ideal, works fairly well. But if you are stepping into GB of data, you may need to chunk them up and reassemble. On Friday, March 4, 2016, Mahesh Dharmasena wrote: > We have a client with several thousand stores which send and receive > messages to m

Kafka Applicability - Large Messages

We have a client with several thousand stores which send and receive messages to main system that resides on the headquarters. A single Store sends and receive around 50 to 100 messages per day. Average Message size could be from 2KB to 1GB. Please let me know whether I can adapt Apache Kafka fo