I think my test include some grok filters and file input so it's not necessarily bottlenecked on Kafka producer.
On Thu, Mar 5, 2015 at 12:37 AM, Vineet Mishra <clearmido...@gmail.com> wrote: > Hey Roger, > > As per your stats you have around 5k msg/s of size 42 bytes > > 5000msgs * 42 byte = 210000 = ~ 205kbps > > while I am getting around 500 msgs of around 350 bytes. > > 500msgs * 350 = 175000 = ~ 170kbps > > Which is even collectively very degrading write throughput. > > It seems this rate of publishing of data is quiet slow with Logstash, as > the Kafka claims for write throughput to be around 50Mbps(please refer this > link http://kafka.apache.org/07/performance.html ) which obviously be the > best performance with the config. tuning and infact I have myself have > achieved around 6-10Mbps of data transfer while publishing to the kafka > using Kafka client. > > I guess this latency is their at Logstash end and perhaps we need to look > for an alternative to the same. > > Do let me know your observation and understanding as well. > > Thanks! > > > > On Thu, Mar 5, 2015 at 1:13 PM, Roger Hoover <roger.hoo...@gmail.com> > wrote: > > > Seeing around 5k msgs/s. The messages are small (average 42 bytes after > > snappy compression) > > > > On Wed, Mar 4, 2015 at 11:34 PM, Vineet Mishra <clearmido...@gmail.com> > > wrote: > > > > > Hi Roger, > > > > > > I have already enabled the snappy, the throughput which I have > mentioned > > is > > > after only. > > > > > > Could you mention what's the throughput you have reaching. > > > > > > Thanks! > > > > > > On Thu, Mar 5, 2015 at 12:56 PM, Roger Hoover <roger.hoo...@gmail.com> > > > wrote: > > > > > > > Hi Vineet, > > > > > > > > Try enabling compression. That improves throughput 3-4x usually for > > me. > > > > Also, you can use async mode if you're willing to trade some chance > of > > > > dropping messages for more throughput. > > > > > > > > kafka { > > > > > > > > codec => 'json' > > > > > > > > broker_list => "localhost:9092" > > > > > > > > topic_id => "blah" > > > > > > > > compression_codec => "snappy" > > > > > > > > request_required_acks => 1 > > > > > > > > producer_type => "async" > > > > > > > > } > > > > > > > > On Wed, Mar 4, 2015 at 11:03 PM, Vineet Mishra < > clearmido...@gmail.com > > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I am having a Logstash Forwarder which is publishing events to > Kafka, > > > but > > > > > as I can see the rate at which the events is published to Kafka is > > > really > > > > > very slow. > > > > > > > > > > With the reference to some links I could get the Kafka Publish > > > throughput > > > > > reaching in 50-60Mbs per second but in my case I am hardly getting > > few > > > > Kbs' > > > > > of throughput. > > > > > > > > > > Looking for Expert intervention to the same. > > > > > > > > > > Thanks, > > > > > > > > > > > > > > >
