Re: Custom File Sink using EventTime and defined custom file name for parquet file

Oops, sorry for not sending the reply to everyone and thanks David for reposting it here. Great to hear that you solved your issue! Kostas On Wed, Jan 15, 2020 at 1:57 PM David Magalhães wrote: > > Sorry, I've only saw the replies today. > > Regarding my previous email, > >> Still, there is so

Re: Custom File Sink using EventTime and defined custom file name for parquet file

Sorry, I've only saw the replies today. Regarding my previous email, Still, there is something missing in this solution to close a window for > with a giving timeout, so it can write into the sink the last events if no > more events are sent. I've fixed this using a custom trigger, val flag =

Re: Custom File Sink using EventTime and defined custom file name for parquet file

Hi David, I'm pulling in Kostas who worked on the StreamingFileSink and might be able to answer some of your questions. Cheers, Till On Mon, Jan 13, 2020 at 2:45 PM Leonard Xu wrote: > Hi, David > > For you first description, I’m a little confused about duplicated records > when backfilling, c

Re: Custom File Sink using EventTime and defined custom file name for parquet file

Hi, David For you first description, I’m a little confused about duplicated records when backfilling, could you describe your usage scenario/code more? I remembered a backfill user solution from Pinterest which is very similar to yours and using Flink too[1], hope that can help you. Best, Leo

Custom File Sink using EventTime and defined custom file name for parquet file

Hi, I'm working for the first time with Flink and I'm trying to create solution that will store events from Kafka into Parquet files in S3. This also should support re-injection of events from Parquet files into a Kafka topic. Here