Roger,
Consider using rsyslog with omkafka. rsyslog rocks! And it's pretty
popular, too -
http://blog.sematext.com/2014/10/06/top-5-most-popular-log-shippers/ Oh,
and it's FAST - some numbers and charts with an older version from 1 year
ago:
http://blog.sematext.com/2014/01/20/rsyslog-8-1-elasti
I think my test include some grok filters and file input so it's not
necessarily bottlenecked on Kafka producer.
On Thu, Mar 5, 2015 at 12:37 AM, Vineet Mishra
wrote:
> Hey Roger,
>
> As per your stats you have around 5k msg/s of size 42 bytes
>
> 5000msgs * 42 byte = 21 = ~ 205kbps
>
> whil
Hey Roger,
As per your stats you have around 5k msg/s of size 42 bytes
5000msgs * 42 byte = 21 = ~ 205kbps
while I am getting around 500 msgs of around 350 bytes.
500msgs * 350 = 175000 = ~ 170kbps
Which is even collectively very degrading write throughput.
It seems this rate of publishi
Seeing around 5k msgs/s. The messages are small (average 42 bytes after
snappy compression)
On Wed, Mar 4, 2015 at 11:34 PM, Vineet Mishra
wrote:
> Hi Roger,
>
> I have already enabled the snappy, the throughput which I have mentioned is
> after only.
>
> Could you mention what's the throughput
Hi Roger,
I have already enabled the snappy, the throughput which I have mentioned is
after only.
Could you mention what's the throughput you have reaching.
Thanks!
On Thu, Mar 5, 2015 at 12:56 PM, Roger Hoover
wrote:
> Hi Vineet,
>
> Try enabling compression. That improves throughput 3-4x u
Hi Vineet,
Try enabling compression. That improves throughput 3-4x usually for me.
Also, you can use async mode if you're willing to trade some chance of
dropping messages for more throughput.
kafka {
codec => 'json'
broker_list => "localhost:9092"
topic_id => "blah"
