Thanks very much for the detailed explanation Stefan.
Regards, Raja. From: Stefan Richter <s.rich...@data-artisans.com> Date: Monday, August 14, 2017 at 7:47 AM To: Raja Aravapalli <raja.aravapa...@target.com> Cc: "user@flink.apache.org" <user@flink.apache.org> Subject: Re: [EXTERNAL] difference between checkpoints & savepoints Just noticed that I forgot to include also a reference to the documentation about externalized checkpoints: https://ci.apache.org/projects/flink/flink-docs-master/ops/state/checkpoints.html Am 14.08.2017 um 14:17 schrieb Stefan Richter <s.rich...@data-artisans.com<mailto:s.rich...@data-artisans.com>>: Hi, Also, in the same line, can someone detail the difference between State Backend & External checkpoint? Those are two very different things. If we talk about state backends in Flink, we mean the entity that is responsible for storing and managing the state inside an operator. This could for example be something like the FsStateBackend that is based on hash maps and keeps state on the heap, or the RocksDBStateBackend which is using RocksDB as a store internally and operates on native memory and disk. An externalized checkpoint, like a normal checkpoint, is the collection of all state in a job persisted to stable storage for recovery. A little more concrete, this typically means writing out the contents of the state backends to a save place so that we can restore them from there. Also, programmatic API, thru which methods we can configure those. This explains how to set the backend programatically: https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html To activate externalized checkpoints, you activate normal checkpoints, plus the following line: env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION); where env is your StreamExecutionEnvironment. If you need an example, please take a look at the org.apache.flink.test.checkpointing.ExternalizedCheckpointITCase. This class configures everything you asked about programatically. Best, Stefan
