Hi Sharninder,
Thanks for the response. The load balancing is not based on header. To
simplify, lets say I have one web server generating logs and three flume
nodes receiving those logs. I want the load to be balanced on those three
flume nodes based on cpu utilization and load.
On Thu, Aug 14, 2014 at 4:01 PM, Sharninder <sharnin...@gmail.com> wrote:
> To add headers to the events, you can either send proper avro formatted
> packets (which have a header) to an avro source, or implement a custom
> interceptor to add headers after they're received by the syslog source.
> There is a static interceptor bundled with flume that you can use. The
> problem with that is that you can only add a single header (key->value) at
> a time, as far as I know. But, its a good starting point to do what you
> want to do.
>
> I didn't really understand your load balancing requirement but if its
> based on the headers, you'll have to write your own interceptors.
>
>
>
> On Thu, Aug 14, 2014 at 12:55 PM, Mohit Durgapal <durgapalmo...@gmail.com>
> wrote:
>
>> I have a requirement where I need to feed push traffic(comma separated
>> logs) at a very high rate to flume.
>> I have three concerns:
>>
>>
>> 1. I am using php to send events to flume through rsyslog. The code I
>> am using is :
>>
>> *openlog("mylogs", LOG_NDELAY, LOG_LOCAL2);
>> syslog(LOG_INFO, "aaid,bid,cid,info1,info2,....");
>> closelog();*
>>
>> I want to add some fields as headers in the above event log "
>> *aaid,bid,cid,info1,info2,....*" , I don't see any function in php
>> where I could add headers for some fields so that I can take some action
>> based on just the headers without opening the complete msg.
>>
>> 2. How to load balance the trafffic. I want the logger to forward the
>> logs to the load balancer and then the load balancer to choose a flume
>> node(based on various factors like current load, cpu utilization) and also
>> handle failures(divert traffic if a flume node goes down).
>>
>> I looked at the flume based load balancer but it provides just two
>> options: Round Robin and Random load balancing. Any ideas as to how I
>> could
>> do this load balancing with failure detection and handling would be very
>> helpful.
>>
>> 3. I want to update a cache in real-time from flume(using
>> interceptor). I want a hashing based approach to divert certain
>> traffic(based on a field or header in log) to certain nodes, so that one
>> node is responsible for updating rows with keys under same hash bucket.
>> This is to avoid row level locking.
>>
>>
>> I hope I have explained my requirements well enough for everyone to
>> understand. But If it's not as clear as I think, please let me know.
>>
>>
>> Regards
>> Mohit
>>
>
>
