Re: Flume regex extractor interceptor

2018-04-20 Thread Ferenc Szabo
Hi Jyotsna, the Regex Extractor Interceptor reads only the event body and writes into the headers. However, you can write custom interceptor easily by implementing this interface: https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/interceptor/Interceptor.jav

Re: Flume regex extractor interceptor

2018-04-20 Thread Jyotsna G
Thank you so much. Implemented a custom interceptor as suggested. Regards, Jyotsna On Fri, Apr 20, 2018 at 2:06 PM, Ferenc Szabo wrote: > Hi Jyotsna, > > the Regex Extractor Interceptor reads only the event body and writes into > the headers. > > However, you can write custom interceptor easil

Re: Flume spooldir source- subdirectory issue

2018-04-20 Thread Ferenc Szabo
Hi Jyotsna, I guess this issue is solved by the custom interceptor mentioned in your other mail. let me know if You need any other help. Regards, Ferenc On Thu, Apr 19, 2018 at 5:57 AM, Jyotsna G wrote: > Hi All, > > I am currently using Flume 1.7 . Configured a spooling directory source. I >

Append existing Avro file - HDFS Sink

2018-04-20 Thread Nitin Kumar
Hi All, I am using Flume v1.8 in which Flume agent comprises of Kafka Channel & HDFS Sink. I am able to write data in Avro file on HDFS into a external HIVE table, but the problem is whenever Flume gets restarted it closes that file and open a new file because of which I can see many small files.

Re: Append existing Avro file - HDFS Sink

2018-04-20 Thread Matt Sicker
It's not a Flume native solution, but an alternative I used in the past was Kafka Connect using the HDFS connector plugin. That plugin provides configuration regarding how often to roll over Avro files. On 20 April 2018 at 13:49, Nitin Kumar wrote: > Hi All, > > I am using Flume v1.8 in which Fl