Hey Roger, As per your stats you have around 5k msg/s of size 42 bytes
5000msgs * 42 byte = 210000 = ~ 205kbps while I am getting around 500 msgs of around 350 bytes. 500msgs * 350 = 175000 = ~ 170kbps Which is even collectively very degrading write throughput. It seems this rate of publishing of data is quiet slow with Logstash, as the Kafka claims for write throughput to be around 50Mbps(please refer this link http://kafka.apache.org/07/performance.html ) which obviously be the best performance with the config. tuning and infact I have myself have achieved around 6-10Mbps of data transfer while publishing to the kafka using Kafka client. I guess this latency is their at Logstash end and perhaps we need to look for an alternative to the same. Do let me know your observation and understanding as well. Thanks! On Thu, Mar 5, 2015 at 1:13 PM, Roger Hoover <roger.hoo...@gmail.com> wrote: > Seeing around 5k msgs/s. The messages are small (average 42 bytes after > snappy compression) > > On Wed, Mar 4, 2015 at 11:34 PM, Vineet Mishra <clearmido...@gmail.com> > wrote: > > > Hi Roger, > > > > I have already enabled the snappy, the throughput which I have mentioned > is > > after only. > > > > Could you mention what's the throughput you have reaching. > > > > Thanks! > > > > On Thu, Mar 5, 2015 at 12:56 PM, Roger Hoover <roger.hoo...@gmail.com> > > wrote: > > > > > Hi Vineet, > > > > > > Try enabling compression. That improves throughput 3-4x usually for > me. > > > Also, you can use async mode if you're willing to trade some chance of > > > dropping messages for more throughput. > > > > > > kafka { > > > > > > codec => 'json' > > > > > > broker_list => "localhost:9092" > > > > > > topic_id => "blah" > > > > > > compression_codec => "snappy" > > > > > > request_required_acks => 1 > > > > > > producer_type => "async" > > > > > > } > > > > > > On Wed, Mar 4, 2015 at 11:03 PM, Vineet Mishra <clearmido...@gmail.com > > > > > wrote: > > > > > > > Hi, > > > > > > > > I am having a Logstash Forwarder which is publishing events to Kafka, > > but > > > > as I can see the rate at which the events is published to Kafka is > > really > > > > very slow. > > > > > > > > With the reference to some links I could get the Kafka Publish > > throughput > > > > reaching in 50-60Mbs per second but in my case I am hardly getting > few > > > Kbs' > > > > of throughput. > > > > > > > > Looking for Expert intervention to the same. > > > > > > > > Thanks, > > > > > > > > > >
