Re: how to load balance flume

2014-08-14 Thread Sharninder
I'm not sure without looking at the exact usecase, but maybe you can use something like haproxy? -- Sharninder On Thu, Aug 14, 2014 at 4:08 PM, Mohit Durgapal wrote: > Hi Sharninder, > > Thanks for the response. The load balancing is not based on header. To > simplify, lets say I have one we

Re: how to load balance flume

2014-08-14 Thread Mohit Durgapal
Hi Sharninder, Thanks for the response. The load balancing is not based on header. To simplify, lets say I have one web server generating logs and three flume nodes receiving those logs. I want the load to be balanced on those three flume nodes based on cpu utilization and load. On Thu, Aug 1

Re: how to load balance flume

2014-08-14 Thread Sharninder
To add headers to the events, you can either send proper avro formatted packets (which have a header) to an avro source, or implement a custom interceptor to add headers after they're received by the syslog source. There is a static interceptor bundled with flume that you can use. The problem with