Hi there, Firstly, a note that Kafka isn't really designed for this kind of large message. http://ingest.tips/2015/01/21/handling-large-messages-kafka/ covers a lot of tips around this use case however, and covers some tuning that will likely improve your usage.
In particular, I expect tuning up fetch.message.max.bytes on the consumer to help out a lot here. Generally though, doing large messages will lead to very low throughput and lots of stability issues, as noted in that article. We run thousands of clusters in production, and typically recommend folk keep message sizes down to the few tens of KB for most use cases. Thanks Tom Crayford Heroku Kafka On Wed, Jun 1, 2016 at 9:49 PM, prateek arora <prateek.arora...@gmail.com> wrote: > I have 4 node kafka broker with following configuration : > > Default Number of Partitions : num.partitions : 1 > Default Replication Factor : default.replication.factor : 1 > Maximum Message Size : message.max.bytes : 10 MB > Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB > > > Right now I have 4 topic with 1 partition and 1 replication factor . > > "Topic Name" : "Broker Id" : "Total Messages Received Across Kafka > Broker" : "Total Bytes Received Across Kafka Broker" > Topic 1 - Leader Kafka Broker 1 : 4.67 Message/Second : 1.6 MB/second > Topic 2 - Leader Kafka Broker 2 : 4.78 Message/Second : 4.1 MB/second > Topic 3 - Leader Kafka Broker 1 : 4.83 Message/Second : 1.6 MB/second > Topic 4 - Leader Kafka Broker 3 : 4.8 Message/Second : 4.3 MB/second > > Message consist of <String , Array[Byte] >. > > > when consumer tried to read message from "Topic 2" Kafka Broker rate of > message receiving slow down from 4.77 message/second to 3.12 > message/second , after some time try to goes up . > > I also attached screenshot of "Total Messages Received Across Kafka > Broker" and "Total Bytes Received Across Kafka Broker" for topic "Topic > 2" . > > can someone explain why it is happen and how to solve it ? > > Regards > Prateek > > > > > > >
