one-to-many interceptor

I see in the documentation for org.apache.flume.interceptor.Interceptor that the result of intercept(List) must not exceed the size of the input (in all-caps, even). This is unfortunate for my use-case: I'm interfacing with a scribe source that provides each message as a serialization of some numbe

Re: one-to-many interceptor

Hi Matt, If you can guarantee there are a certain # of events in a single "wrapper" event, or bound the limit, then you could potentially get away with this. However if you're not careful you could get stuck in an infinite fail-backoff-retry loop due to exceeding the (configurable) channel transact

Re: flume starting through service

Can you please provide details on the errors you are seeing? What version of Flume? On Thu, Jun 19, 2014 at 12:43 AM, kishore alajangi < alajangikish...@gmail.com> wrote: > > The flume is writing to hdfs when I start flume manually through config > file like > > flume-ng agent -c /etc/flume-ng/c

Re: one-to-many interceptor

Thanks Mike. That's very helpful. It should be no trouble to establish an upper bound on the intra-event packing, so I should be able to tune the capacity parameters in the way you suggest. Besides, aren't the sharp edges what make it fun to play with? I may be able to impose suitable restriction

Re: flume starting through service

Flume 1.4.0-cdh4.5.0 On Wed, Jun 25, 2014 at 7:12 AM, Mike Percy wrote: > Can you please provide details on the errors you are seeing? What version > of Flume? > > > On Thu, Jun 19, 2014 at 12:43 AM, kishore alajangi < > alajangikish...@gmail.com> wrote: > >> >> The flume is writing to hdfs whe

Re: one-to-many interceptor

Matt, no prob! Mike Sent from my iPhone > On Jun 24, 2014, at 8:35 PM, Matt Tenenbaum > wrote: > > Thanks Mike. That's very helpful. > > It should be no trouble to establish an upper bound on the intra-event > packing, so I should be able to tune the capacity parameters in the way you > su

Re: flume starting through service

Ok so what's the error? Mike Sent from my iPhone > On Jun 24, 2014, at 10:44 PM, kishore alajangi > wrote: > > Flume 1.4.0-cdh4.5.0 > > >> On Wed, Jun 25, 2014 at 7:12 AM, Mike Percy wrote: >> Can you please provide details on the errors you are seeing? What version of >> Flume? >> >> >

Re: flume starting through service

No error, my view is when i start #service flume-ng-agent it will work as same as flume-ng agent -c /etc/flume-ng/conf -f /etc/flume-ng/conf/flume.conf -n agent. but no result in the output. On Wed, Jun 25, 2014 at 10:27 AM, Mike Percy wrote: > Ok so what's the error? > > Mike > > Sent from my

Re: flume starting through service

Check the log which is at /var/log/flume-ng/flume.log Mike Sent from my iPhone > On Jun 24, 2014, at 11:36 PM, kishore alajangi > wrote: > > No error, my view is when i start #service flume-ng-agent it will work as > same as flume-ng agent -c /etc/flume-ng/conf -f /etc/flume-ng/conf/flume.co

spooldir to hdfs

Hi Flume Experts, Could anybody help me to store avro files located in my localfilesystem into hdfs using flume, frequently new files will be generated so "spooldir" work for it as source, -- Thanks, Kishore.