Re: Questions about checkpoints/savepoints

Hi team, I am a similar use case do we have any answers on this? When we trigger savepoint can we store that information to ZK as well? So I can avoid S3 file listing and do not have to use other external services? On Wed, Oct 25, 2017 at 11:19 PM vipul singh wrote: > As a followup to above, is

Re: Questions about checkpoints/savepoints

As a followup to above, is there a way to get the last checkpoint metadata location inside *notifyCheckpointComplete* method? I tried poking around, but didnt see a way to achieve this. Or incase there is any other way to save the actual checkpoint metadata location information into a datastore(dy

Re: Questions about checkpoints/savepoints

Thanks Aljoscha for the explanations. I was able to recover from the last externalized checkpoint, by using flink run -s I am curious, are there any options to save the metadata file name to some other place like dynamo etc at the moment? The reason why I am asking is, for the end launcher code

Re: Questions about checkpoints/savepoints

Hi, That distinction with externalised checkpoints is a bit of a pitfall and I'm hoping that we can actually get rid of that distinction in the next version or the version after that. With that change, all checkpoints would always be externalised, since it's not really any noticeable overhead.

Re: Questions about checkpoints/savepoints

Thanks Tony, that was the issue. I was thinking that when we use Rocksdb and provide an s3 path, it uses externalized checkpoints by default. Thanks so much! I have one followup question. Say in above case, I terminate the cluster, and since the metadata is on s3, and not on local storage, does fl

Re: Questions about checkpoints/savepoints

Hi, Did you enable externalized checkpoints? [1] Best, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/checkpoints.html#externalized-checkpoints 2017-10-24 13:07 GMT+08:00 vipul singh : > Thanks Aljoscha for the answer above. > > I am experimenting with savepoint

Re: Questions about checkpoints/savepoints

Thanks Aljoscha for the answer above. I am experimenting with savepoints and checkpoints on my end, so that we built fault tolerant application with exactly once semantics. I have been able to test various scenarios, but have doubts about one use case. My app is running on an emr cluster, and I

Re: Questions about checkpoints/savepoints

Hi, Flink does not rely on file system operations to list contents, all necessary file paths are stored in the meta data file, as you guessed. This is the reason savepoints also work with file systems that "only" have read-after-write consistency. Best, Aljoscha > On 10. Oct 2017, at 03:01, v

Re: Questions about checkpoints/savepoints

Thanks Stefan for the answers above. These are really helpful. I have a few followup questions: 1. I see my savepoints are created in a folder, which has a _metadata file and another file. Looking at the code

Re: Questions about checkpoints/savepoints

Hi, I have answered your questions inline: > It seems to me that checkpoints can be treated as flink internal recovery > mechanism, and savepoints act more as user-defined recovery points. Would > that be a correct assumption? You could see it that way, but I would describe savepoints more as us

Questions about checkpoints/savepoints

Hello, I have some confusion about checkpoints vs savepoints, and how to use them effectively in my application. I am working on an application which is relies on flink's fault tolerant mechanism to ensure exactly once semantics. I have enabled external checkpointing in my application as below: