low-latency solution from avro to parquet

2017-04-12 Thread big data
anybody know how to use flume to move avro source data to parquet destination? And is there a low-latency solution to change avro file to parquet? thanks.

TAILDIR source causes many channel closed error

2017-11-29 Thread big data
I use TAILDIR sink in flume 1.7 to catch file changed in some folders and sink to HDFS, but after start flume, some source runs well, some source causes Channel-closed error. ERROR [PollableSourceRunner-TaildirSource-source-error] (org.apache.flume.source.taildir.TaildirSource.process:236) -Una

How to get the status of file as Taildir source

2021-01-19 Thread big data
Dear all, We set a folder as taildir source, and there are many files will be added every day in this folder. After flume TailDir source read these files, position info was recorded in internal. How we can knows that has one file been read totally or not?