Re: Flink recovery

2016-05-17 Thread Fabian Hueske
er > Reply-To: "user@flink.apache.org" > Date: Tuesday, May 17, 2016 at 10:02 AM > > To: "user@flink.apache.org" > Subject: Re: Flink recovery > > Hi, > > Savepoints are exactly for that use case: > https://ci.apache.org/projects/flink/flink-docs

Re: Flink recovery

2016-05-17 Thread Madhire, Naveen
gt;" mailto:user@flink.apache.org>> Date: Tuesday, May 17, 2016 at 10:02 AM To: "user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Subject: Re: Flink recovery Hi, Savepoints are exactly for that use case: https://ci.apache.o

Re: Flink recovery

2016-05-17 Thread Robert Metzger
e, application > upgrade and node failure. > > > > Thanks, > Naveen > > From: Robert Metzger > Reply-To: "user@flink.apache.org" > Date: Tuesday, May 17, 2016 at 6:58 AM > To: "user@flink.apache.org" > Subject: Re: Flink recovery >

Re: Flink recovery

2016-05-17 Thread Madhire, Naveen
Thank you. From: Fabian Hueske mailto:fhue...@gmail.com>> Reply-To: "user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Date: Saturday, May 14, 2016 at 4:10 AM To: "user@flink.apache.org<mailto:user@flink.apache.org>" mailto:us

Re: Flink recovery

2016-05-17 Thread Robert Metzger
>> Thank you. >> >> From: Fabian Hueske >> Reply-To: "user@flink.apache.org" >> Date: Saturday, May 14, 2016 at 4:10 AM >> To: "user@flink.apache.org" >> Subject: Re: Flink recovery >> >> The behavior of the RollingFileSink de

Re: Flink recovery

2016-05-17 Thread Stephan Ewen
link.apache.org" > Date: Saturday, May 14, 2016 at 4:10 AM > To: "user@flink.apache.org" > Subject: Re: Flink recovery > > The behavior of the RollingFileSink depends on the capabilities of the > file system. > If the file system does not support to truncate

Re: Flink recovery

2016-05-16 Thread Madhire, Naveen
"user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Date: Saturday, May 14, 2016 at 4:10 AM To: "user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Subject: Re: Flink recovery The behavior of the

Re: Flink recovery

2016-05-14 Thread Fabian Hueske
t; pipeline. > > > > Thanks, > Naveen > > From: Fabian Hueske > Reply-To: "user@flink.apache.org" > Date: Friday, May 13, 2016 at 4:26 PM > > To: "user@flink.apache.org" > Subject: Re: Flink recovery > > Hi Naveen, > > the Ro

Re: Flink recovery

2016-05-13 Thread Madhire, Naveen
gt;" mailto:user@flink.apache.org>> Date: Friday, May 13, 2016 at 4:26 PM To: "user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Subject: Re: Flink recovery Hi Naveen, the RollingFileSink supports exactly-once output. So you should be good

Re: Flink recovery

2016-05-13 Thread Fabian Hueske
Reply-To: "user@flink.apache.org" > Date: Friday, May 13, 2016 at 4:13 PM > To: "user@flink.apache.org" > Subject: Re: Flink recovery > > Hi, > > Flink's exactly-once semantics do not mean that events are processed > exactly-once but that events

Re: Flink recovery

2016-05-13 Thread Madhire, Naveen
gt;" mailto:user@flink.apache.org>> Date: Friday, May 13, 2016 at 4:13 PM To: "user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Subject: Re: Flink recovery Hi, Flink's exactly-once semantics do not mean that events are processed exact

Re: Flink recovery

2016-05-13 Thread Fabian Hueske
Hi, Flink's exactly-once semantics do not mean that events are processed exactly-once but that events will contribute exactly-once to the state of an operator such as a counter. Roughly, the mechanism works as follows: - Flink peridically injects checkpoint markers into the data stream. This happe

Re: Flink recovery

2016-05-13 Thread Madhire, Naveen
I checked the JIRA and looks like FLINK-2111 should address the issue which I am facing. I am canceling the job from dashboard. I am using kafka source and HDFS rolling sink. https://issues.apache.org/jira/browse/FLINK-2111 Is this JIRA part of Flink 1.0.0? Thanks, Naveen From: "Madhire, Ve