Query regarding flume spooling directory Source fileSuffix

Hi, Am currently using flume 1.7.0 . Configured spooling directory source and writing files in to HDFS sink. The source directory contains csv files which need to be written to HDFS, these csv files are also read by other applications. Is there a way I can avoid adding a fileSuffix to these csv f

Flume spooldir source- subdirectory issue

Hi All, I am currently using Flume 1.7 . Configured a spooling directory source. I have enabled recursiveDirectorySearch=true to look in to the sub directories for files. source.spoolDir=/tmp/test and under /tmp/test, subdirectories get created with data files /tmp/test/data1/file.csv , /tmp/tes

Flume regex extractor interceptor

Hi All, Will flume regex extractor interceptor be applied on flume even header values also or only the body? Can i use it to extract a substring out of the fileHeader value in the event header? Thanks, Jyotsna

Re: Flume regex extractor interceptor

custom interceptor easily by implementing this > interface: > > https://github.com/apache/flume/blob/trunk/flume-ng- > core/src/main/java/org/apache/flume/interceptor/Interceptor.java > > as you can see, you get the whole Event. > > Regards, > Ferenc > > On Fri, Apr

Issue similar to Flume-2052

Hi All, Am currently using Flume 1.7.0. While processing a set of files, am encountering the below exception and Flume hangs unpredictably. 18/06/21 05:40:40 ERROR source.SpoolDirectorySource: FATAL: Spool Directory source src-tpa_idoru: { spoolDir: /tmp/tpa-idoru/ }: Uncaught exception in SpoolD

Re: Issue similar to Flume-2052

fault it is FAIL, but it must be IGNORE or REPLACE in order to > proceed further when decoding error occurs. > > Regrads, > Peter Turcsanyi > > On Thu, Jun 21, 2018 at 2:54 PM, Jyotsna G wrote: > >> Hi All, >> Am currently using Flume 1.7.0. >> >> While proc

Query regarding customizing baseNameHeader value

Hello, I have a spoolDir flume agent configured and the source path contains ":" colon character in it which results in the below exception while writing in to HDFS. org.apache.flume.EventDeliveryException: java.lang.IllegalArgumentException: Pathname /TPA_PAAS/20180626.01:21:14/IDCS-CertificateA

Re: Query regarding customizing baseNameHeader value

rs in > the event header fields (similar to SearchAndReplaceInterceptor, but this > works only with the event body). > > Regards, > Peter Turcsanyi > > > On Wed, Jun 27, 2018 at 2:00 PM, Jyotsna G wrote: > >> Hello, >> >> I have a spoolDir flume agent confi