Seeing around 5k msgs/s. The messages are small (average 42 bytes after
snappy compression)
On Wed, Mar 4, 2015 at 11:34 PM, Vineet Mishra <clearmido...@gmail.com>
wrote:
> Hi Roger,
>
> I have already enabled the snappy, the throughput which I have mentioned is
> after only.
>
> Could you mention what's the throughput you have reaching.
>
> Thanks!
>
> On Thu, Mar 5, 2015 at 12:56 PM, Roger Hoover <roger.hoo...@gmail.com>
> wrote:
>
> > Hi Vineet,
> >
> > Try enabling compression. That improves throughput 3-4x usually for me.
> > Also, you can use async mode if you're willing to trade some chance of
> > dropping messages for more throughput.
> >
> > kafka {
> >
> > codec => 'json'
> >
> > broker_list => "localhost:9092"
> >
> > topic_id => "blah"
> >
> > compression_codec => "snappy"
> >
> > request_required_acks => 1
> >
> > producer_type => "async"
> >
> > }
> >
> > On Wed, Mar 4, 2015 at 11:03 PM, Vineet Mishra <clearmido...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I am having a Logstash Forwarder which is publishing events to Kafka,
> but
> > > as I can see the rate at which the events is published to Kafka is
> really
> > > very slow.
> > >
> > > With the reference to some links I could get the Kafka Publish
> throughput
> > > reaching in 50-60Mbs per second but in my case I am hardly getting few
> > Kbs'
> > > of throughput.
> > >
> > > Looking for Expert intervention to the same.
> > >
> > > Thanks,
> > >
> >
>
