I think my test include some grok filters and file input so it's not
necessarily bottlenecked on Kafka producer.

On Thu, Mar 5, 2015 at 12:37 AM, Vineet Mishra <clearmido...@gmail.com>
wrote:

> Hey Roger,
>
> As per your stats you have around 5k msg/s of size 42 bytes
>
> 5000msgs * 42 byte = 210000 = ~ 205kbps
>
> while I am getting  around 500 msgs of around 350 bytes.
>
> 500msgs * 350 = 175000 = ~ 170kbps
>
> Which is even collectively very degrading write throughput.
>
> It seems this rate of publishing of data is quiet slow with Logstash, as
> the Kafka claims for write throughput to be around 50Mbps(please refer this
> link http://kafka.apache.org/07/performance.html ) which obviously be the
> best performance with the config. tuning and infact I have myself have
> achieved around 6-10Mbps of data transfer while publishing to the kafka
> using Kafka client.
>
> I guess this latency is their at Logstash end and perhaps we need to look
> for an alternative to the same.
>
> Do let me know your observation and understanding as well.
>
> Thanks!
>
>
>
> On Thu, Mar 5, 2015 at 1:13 PM, Roger Hoover <roger.hoo...@gmail.com>
> wrote:
>
> > Seeing around 5k msgs/s.  The messages are small (average 42 bytes after
> > snappy compression)
> >
> > On Wed, Mar 4, 2015 at 11:34 PM, Vineet Mishra <clearmido...@gmail.com>
> > wrote:
> >
> > > Hi Roger,
> > >
> > > I have already enabled the snappy, the throughput which I have
> mentioned
> > is
> > > after only.
> > >
> > > Could you mention what's the throughput you have reaching.
> > >
> > > Thanks!
> > >
> > > On Thu, Mar 5, 2015 at 12:56 PM, Roger Hoover <roger.hoo...@gmail.com>
> > > wrote:
> > >
> > > > Hi Vineet,
> > > >
> > > > Try enabling compression.  That improves throughput 3-4x usually for
> > me.
> > > > Also, you can use async mode if you're willing to trade some chance
> of
> > > > dropping messages for more throughput.
> > > >
> > > >     kafka {
> > > >
> > > >       codec => 'json'
> > > >
> > > >       broker_list => "localhost:9092"
> > > >
> > > >       topic_id => "blah"
> > > >
> > > >       compression_codec => "snappy"
> > > >
> > > >       request_required_acks => 1
> > > >
> > > >       producer_type => "async"
> > > >
> > > >     }
> > > >
> > > > On Wed, Mar 4, 2015 at 11:03 PM, Vineet Mishra <
> clearmido...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am having a Logstash Forwarder which is publishing events to
> Kafka,
> > > but
> > > > > as I can see the rate at which the events is published to Kafka is
> > > really
> > > > > very slow.
> > > > >
> > > > > With the reference to some links I could get the Kafka Publish
> > > throughput
> > > > > reaching in 50-60Mbs per second but in my case I am hardly getting
> > few
> > > > Kbs'
> > > > > of throughput.
> > > > >
> > > > > Looking for Expert intervention to the same.
> > > > >
> > > > > Thanks,
> > > > >
> > > >
> > >
> >
>

Reply via email to