Re: Issues in recovering state from last crash using custom sink

2017-08-29 Thread Aljoscha Krettek
Hi, If you are not manually doing a savepoint and then restoring from that savepoint you will not restore state. Simply stopping a job and then restarting will not restore state. The regular checkpoints are only used for recovery if a job fails, not for a user-induced shutdown. Best, Aljoscha

Re: Issues in recovering state from last crash using custom sink

2017-08-28 Thread vipul singh
Hi Aljoscha, Yes. I am running the application till a few checkpoints are complete. I am stopping the application between two checkpoints, so there will be messages in the list state, which should be checkpointed when *snapshot* is called. I am able to see a checkpoint file on S3( I am saving the

Re: Issues in recovering state from last crash using custom sink

2017-08-28 Thread Aljoscha Krettek
Hi, How are you testing the recovery behaviour? Are you taking a savepoint ,then shutting down, and then restarting the Job from the savepoint? Best, Aljoscha > On 28. Aug 2017, at 00:28, vipul singh wrote: > > Hi all, > > I am working on a flink archiver application. In a gist this applicat

Issues in recovering state from last crash using custom sink

2017-08-27 Thread vipul singh
Hi all, I am working on a flink archiver application. In a gist this application tries to reads a bunch of schematized messages from kafka and archives them to s3. Due to the nature of the naming of the files, I had to go towards a custom sink implementation. As of the current progress the applica