Re: Question about flink checkpoint

2018-02-08 Thread Fabian Hueske
Great, thank you! Best, Fabian 2018-02-07 23:52 GMT+01:00 Chengzhi Zhao : > Thanks, Fabian, > > I opened an JIRA ticket and I'd like to work on it if people think this > would be a improvement: > https://issues.apache.org/jira/browse/FLINK-8599 > > Best, > Chengzhi > > On Wed, Feb 7, 2018 at 4:1

Re: Question about flink checkpoint

2018-02-07 Thread Chengzhi Zhao
Thanks, Fabian, I opened an JIRA ticket and I'd like to work on it if people think this would be a improvement: https://issues.apache.org/jira/browse/FLINK-8599 Best, Chengzhi On Wed, Feb 7, 2018 at 4:17 AM, Fabian Hueske wrote: > Hi Chengzhi Zhao, > > I think this is rather an issue with the

Re: Question about flink checkpoint

2018-02-07 Thread Fabian Hueske
Hi Chengzhi Zhao, I think this is rather an issue with the ContinuousFileReaderOperator than with the checkpointing algorithm in general. A source can decide which information to store as state and also how to handle failures such as file paths that have been put into state but have been removed f

Fwd: Question about flink checkpoint

2018-02-06 Thread Chengzhi Zhao
Hey, I am new to flink and I have a question and want to see if anyone can help here. So we have a s3 path that flink is monitoring that path to see new files available. val avroInputStream_activity = env.readFile(format, path, FileProcessingMode.PROCESS_CONTINUOUSLY, 1) I am doing both inte