It's not a Flume native solution, but an alternative I used in the past was Kafka Connect using the HDFS connector plugin. That plugin provides configuration regarding how often to roll over Avro files.
On 20 April 2018 at 13:49, Nitin Kumar <nitin.kumar2...@gmail.com> wrote: > Hi All, > > I am using Flume v1.8 in which Flume agent comprises of Kafka Channel & > HDFS Sink. > I am able to write data in Avro file on HDFS into a external HIVE table, > but the problem is whenever Flume gets restarted it closes that file and > open a new file because of which I can see many small files. (Data is > partition on the basis of date) > > Can't Flume append to existing file to avoid creation of new file? > Also, how can I solve this problem which leads to creation of too many > small files? > > Any help would be appreciated. > > -- > > *Regards,Nitin Kumar* > -- Matt Sicker <boa...@gmail.com>
