Re: Flink recovery

2016-05-17 Thread Fabian Hueske
er > Reply-To: "user@flink.apache.org" > Date: Tuesday, May 17, 2016 at 10:02 AM > > To: "user@flink.apache.org" > Subject: Re: Flink recovery > > Hi, > > Savepoints are exactly for that use case: > https://ci.apache.org/projects/flink/flink-docs

Re: Flink recovery

2016-05-17 Thread Madhire, Naveen
gt;" mailto:user@flink.apache.org>> Date: Tuesday, May 17, 2016 at 10:02 AM To: "user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Subject: Re: Flink recovery Hi, Savepoints are exactly for that use case: https://ci.apache.o

Re: Flink recovery

2016-05-17 Thread Robert Metzger
e, application > upgrade and node failure. > > > > Thanks, > Naveen > > From: Robert Metzger > Reply-To: "user@flink.apache.org" > Date: Tuesday, May 17, 2016 at 6:58 AM > To: "user@flink.apache.org" > Subject: Re: Flink recovery >

Re: Flink recovery

2016-05-17 Thread Madhire, Naveen
mailto:user@flink.apache.org>> Subject: Re: Flink recovery Hi Naveen, I think cancelling a job is not the right approach for testing our exactly-once guarantees. By cancelling a job, you are discarding the state of your job. Restarting from scratch (without using a savepoint) will cause dupli

Re: Flink recovery

2016-05-17 Thread Robert Metzger
>> Thank you. >> >> From: Fabian Hueske >> Reply-To: "user@flink.apache.org" >> Date: Saturday, May 14, 2016 at 4:10 AM >> To: "user@flink.apache.org" >> Subject: Re: Flink recovery >> >> The behavior of the RollingFileSink de

Re: Flink recovery

2016-05-17 Thread Stephan Ewen
link.apache.org" > Date: Saturday, May 14, 2016 at 4:10 AM > To: "user@flink.apache.org" > Subject: Re: Flink recovery > > The behavior of the RollingFileSink depends on the capabilities of the > file system. > If the file system does not support to truncate

Re: Flink recovery

2016-05-16 Thread Madhire, Naveen
"user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Date: Saturday, May 14, 2016 at 4:10 AM To: "user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Subject: Re: Flink recovery The behavior of the

Re: Flink recovery

2016-05-14 Thread Fabian Hueske
t; pipeline. > > > > Thanks, > Naveen > > From: Fabian Hueske > Reply-To: "user@flink.apache.org" > Date: Friday, May 13, 2016 at 4:26 PM > > To: "user@flink.apache.org" > Subject: Re: Flink recovery > > Hi Naveen, > > the Ro

Re: Flink recovery

2016-05-13 Thread Madhire, Naveen
gt;" mailto:user@flink.apache.org>> Date: Friday, May 13, 2016 at 4:26 PM To: "user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Subject: Re: Flink recovery Hi Naveen, the RollingFileSink supports exactly-once output. So you should be good

Re: Flink recovery

2016-05-13 Thread Fabian Hueske
Reply-To: "user@flink.apache.org" > Date: Friday, May 13, 2016 at 4:13 PM > To: "user@flink.apache.org" > Subject: Re: Flink recovery > > Hi, > > Flink's exactly-once semantics do not mean that events are processed > exactly-once but that events

Re: Flink recovery

2016-05-13 Thread Madhire, Naveen
gt;" mailto:user@flink.apache.org>> Date: Friday, May 13, 2016 at 4:13 PM To: "user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Subject: Re: Flink recovery Hi, Flink's exactly-once semantics do not mean that events are processed exact

Re: Flink recovery

2016-05-13 Thread Fabian Hueske
een > > From: "Madhire, Venkat Naveen Kumar Reddy" > Reply-To: "user@flink.apache.org" > Date: Friday, May 13, 2016 at 10:58 AM > To: "user@flink.apache.org" > Subject: Flink recovery > > Hi, > > We are trying to test the recovery mechanis

Re: Flink recovery

2016-05-13 Thread Madhire, Naveen
.org>" mailto:user@flink.apache.org>> Subject: Flink recovery Hi, We are trying to test the recovery mechanism of Flink with Kafka and HDFS sink during failures. I’ve killed the job after processing some messages and restarted the same job again. Some of the messages I am seeing

Flink recovery

2016-05-13 Thread Madhire, Naveen
Hi, We are trying to test the recovery mechanism of Flink with Kafka and HDFS sink during failures. I’ve killed the job after processing some messages and restarted the same job again. Some of the messages I am seeing are processed more than once and not following the exactly once semantics.