Hi Thanks for the information . I have one question :
Right now in my scenario maximum message size is around 800KB . did we consider these messages in large size categories , because article told about 10-100 MB data . Regards Prateek On Thu, Jun 2, 2016 at 6:54 AM, Tom Crayford <tcrayf...@heroku.com> wrote: > Hi there, > > Firstly, a note that Kafka isn't really designed for this kind of large > message. http://ingest.tips/2015/01/21/handling-large-messages-kafka/ > covers a lot of tips around this use case however, and covers some tuning > that will likely improve your usage. > > In particular, I expect tuning up fetch.message.max.bytes on the consumer > to help out a lot here. > > Generally though, doing large messages will lead to very low throughput and > lots of stability issues, as noted in that article. We run thousands of > clusters in production, and typically recommend folk keep message sizes > down to the few tens of KB for most use cases. > > Thanks > > Tom Crayford > Heroku Kafka > > On Wed, Jun 1, 2016 at 9:49 PM, prateek arora <prateek.arora...@gmail.com> > wrote: > > > I have 4 node kafka broker with following configuration : > > > > Default Number of Partitions : num.partitions : 1 > > Default Replication Factor : default.replication.factor : 1 > > Maximum Message Size : message.max.bytes : 10 MB > > Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB > > > > > > Right now I have 4 topic with 1 partition and 1 replication factor . > > > > "Topic Name" : "Broker Id" : "Total Messages Received Across Kafka > > Broker" : "Total Bytes Received Across Kafka Broker" > > Topic 1 - Leader Kafka Broker 1 : 4.67 Message/Second : 1.6 MB/second > > Topic 2 - Leader Kafka Broker 2 : 4.78 Message/Second : 4.1 MB/second > > Topic 3 - Leader Kafka Broker 1 : 4.83 Message/Second : 1.6 > MB/second > > Topic 4 - Leader Kafka Broker 3 : 4.8 Message/Second : 4.3 > MB/second > > > > Message consist of <String , Array[Byte] >. > > > > > > when consumer tried to read message from "Topic 2" Kafka Broker rate of > > message receiving slow down from 4.77 message/second to 3.12 > > message/second , after some time try to goes up . > > > > I also attached screenshot of "Total Messages Received Across Kafka > > Broker" and "Total Bytes Received Across Kafka Broker" for topic "Topic > > 2" . > > > > can someone explain why it is happen and how to solve it ? > > > > Regards > > Prateek > > > > > > > > > > > > > > >
