Prateek, hope you looked at compression?

On Thu, Jun 2, 2016 at 10:26 AM, Tom Crayford <tcrayf...@heroku.com> wrote:

> The article says ideal is about 10KB, which holds up well with what we've
> seen in practice as well.
>
> On Thu, Jun 2, 2016 at 6:25 PM, prateek arora <prateek.arora...@gmail.com>
> wrote:
>
> > Hi
> > Thanks for the information .
> >
> > I have one question :
> >
> > Right now in my scenario  maximum message size is around 800KB . did we
> > consider these messages in large size categories , because article told
> > about 10-100 MB data .
> >
> > Regards
> > Prateek
> >
> >
> >
> >
> >
> >
> > On Thu, Jun 2, 2016 at 6:54 AM, Tom Crayford <tcrayf...@heroku.com>
> wrote:
> >
> > > Hi there,
> > >
> > > Firstly, a note that Kafka isn't really designed for this kind of large
> > > message. http://ingest.tips/2015/01/21/handling-large-messages-kafka/
> > > covers a lot of tips around this use case however, and covers some
> tuning
> > > that will likely improve your usage.
> > >
> > > In particular, I expect tuning up fetch.message.max.bytes on the
> consumer
> > > to help out a lot here.
> > >
> > > Generally though, doing large messages will lead to very low throughput
> > and
> > > lots of stability issues, as noted in that article. We run thousands of
> > > clusters in production, and typically recommend folk keep message sizes
> > > down to the few tens of KB for most use cases.
> > >
> > > Thanks
> > >
> > > Tom Crayford
> > > Heroku Kafka
> > >
> > > On Wed, Jun 1, 2016 at 9:49 PM, prateek arora <
> > prateek.arora...@gmail.com>
> > > wrote:
> > >
> > > > I have 4 node kafka broker with following configuration :
> > > >
> > > > Default Number of Partitions  : num.partitions : 1
> > > > Default Replication Factor : default.replication.factor : 1
> > > > Maximum Message Size : message.max.bytes : 10 MB
> > > > Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB
> > > >
> > > >
> > > > Right now I have 4 topic with 1 partition and 1 replication factor .
> > > >
> > > > "Topic Name" : "Broker Id" :  "Total Messages Received Across Kafka
> > > > Broker" : "Total Bytes Received Across Kafka Broker"
> > > > Topic 1  - Leader Kafka Broker 1 :  4.67 Message/Second  :  1.6
> > MB/second
> > > > Topic 2  - Leader Kafka Broker 2 :  4.78 Message/Second  :  4.1
> > MB/second
> > > > Topic 3  - Leader Kafka Broker 1 :  4.83  Message/Second   : 1.6
> > > MB/second
> > > > Topic 4  - Leader Kafka Broker 3  : 4.8 Message/Second   :   4.3
> > > MB/second
> > > >
> > > > Message consist of <String , Array[Byte] >.
> > > >
> > > >
> > > > when consumer tried to read message from "Topic 2"  Kafka Broker rate
> > of
> > > >  message receiving slow down from 4.77 message/second to 3.12
> > > > message/second  , after some time  try to goes up .
> > > >
> > > > I also attached screenshot of "Total Messages Received Across Kafka
> > > > Broker"  and "Total Bytes Received Across Kafka Broker" for topic
> > "Topic
> > > > 2" .
> > > >
> > > > can someone explain why it is happen and how to solve it ?
> > > >
> > > > Regards
> > > > Prateek
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>



-- 
Radha Krishna, Proddaturi
253-234-5657

Reply via email to