Prateek, hope you looked at compression? On Thu, Jun 2, 2016 at 10:26 AM, Tom Crayford <tcrayf...@heroku.com> wrote:
> The article says ideal is about 10KB, which holds up well with what we've > seen in practice as well. > > On Thu, Jun 2, 2016 at 6:25 PM, prateek arora <prateek.arora...@gmail.com> > wrote: > > > Hi > > Thanks for the information . > > > > I have one question : > > > > Right now in my scenario maximum message size is around 800KB . did we > > consider these messages in large size categories , because article told > > about 10-100 MB data . > > > > Regards > > Prateek > > > > > > > > > > > > > > On Thu, Jun 2, 2016 at 6:54 AM, Tom Crayford <tcrayf...@heroku.com> > wrote: > > > > > Hi there, > > > > > > Firstly, a note that Kafka isn't really designed for this kind of large > > > message. http://ingest.tips/2015/01/21/handling-large-messages-kafka/ > > > covers a lot of tips around this use case however, and covers some > tuning > > > that will likely improve your usage. > > > > > > In particular, I expect tuning up fetch.message.max.bytes on the > consumer > > > to help out a lot here. > > > > > > Generally though, doing large messages will lead to very low throughput > > and > > > lots of stability issues, as noted in that article. We run thousands of > > > clusters in production, and typically recommend folk keep message sizes > > > down to the few tens of KB for most use cases. > > > > > > Thanks > > > > > > Tom Crayford > > > Heroku Kafka > > > > > > On Wed, Jun 1, 2016 at 9:49 PM, prateek arora < > > prateek.arora...@gmail.com> > > > wrote: > > > > > > > I have 4 node kafka broker with following configuration : > > > > > > > > Default Number of Partitions : num.partitions : 1 > > > > Default Replication Factor : default.replication.factor : 1 > > > > Maximum Message Size : message.max.bytes : 10 MB > > > > Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB > > > > > > > > > > > > Right now I have 4 topic with 1 partition and 1 replication factor . > > > > > > > > "Topic Name" : "Broker Id" : "Total Messages Received Across Kafka > > > > Broker" : "Total Bytes Received Across Kafka Broker" > > > > Topic 1 - Leader Kafka Broker 1 : 4.67 Message/Second : 1.6 > > MB/second > > > > Topic 2 - Leader Kafka Broker 2 : 4.78 Message/Second : 4.1 > > MB/second > > > > Topic 3 - Leader Kafka Broker 1 : 4.83 Message/Second : 1.6 > > > MB/second > > > > Topic 4 - Leader Kafka Broker 3 : 4.8 Message/Second : 4.3 > > > MB/second > > > > > > > > Message consist of <String , Array[Byte] >. > > > > > > > > > > > > when consumer tried to read message from "Topic 2" Kafka Broker rate > > of > > > > message receiving slow down from 4.77 message/second to 3.12 > > > > message/second , after some time try to goes up . > > > > > > > > I also attached screenshot of "Total Messages Received Across Kafka > > > > Broker" and "Total Bytes Received Across Kafka Broker" for topic > > "Topic > > > > 2" . > > > > > > > > can someone explain why it is happen and how to solve it ? > > > > > > > > Regards > > > > Prateek > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Radha Krishna, Proddaturi 253-234-5657
