The _metadata is always stored in the same directory as the checkpoint data.
As outlined here
<https://ci.apache.org/projects/flink/flink-docs-master/ops/state/checkpoints.html#directory-structure>
"state.checkpoints.dir" serves as a cluster-wide configuration that
_can_ be overwritten with a job-specific setting when creating the
state-backend.
If you want the state-backend to use the configured directory you must
configure the state-backend in the configuration as well, as outlined
here
<https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#setting-default-state-backend>.
On 19/06/2019 16:26, Vishal Sharma wrote:
Hi Folks,
I am using flink 1.8 with externalised checkpointing enabled and
saving the checkpoints to aws S3.
My configuration is as follows :
flink-conf.yaml :
state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata
In application code :
env.setStateBackend(new
RocksDBStateBackend("s3a://test-bucket/checkpoints", true))
As per my understanding, the externalized checkpoint’s meta data is
determined from the configuration key "|state.checkpoints.dir" and
checkpoint data is stored in state backend path. |
|
|
|However, In my case, I don't see anything in the metadata directory.
The _metadata file is present inside each of the checkpoint directory
(chk-6043 ...).
|
|
|
Is this the expected behavior ? If yes, what is the use of
"state.checkpoints.dir" configuration ?
My goal is to establish a process to automatically restart the job
from last completed externalised checkpoint in case of failure. For
this to happen, I need to able to figure out path for the metadata of
latest checkpoint.
Thanks,
Vishal Sharma
*/Grab is hiring. Learn more at //https://grab.careers
<https://grab.careers/>/*
By communicating with Grab Inc and/or its subsidiaries, associate
companies and jointly controlled entities (“Grab Group”), you are
deemed to have consented to the processing of your personal data as
set out in the Privacy Notice which can be viewed at
https://grab.com/privacy/
This email contains confidential information and is only for the
intended recipient(s). If you are not the intended recipient(s),
please do not disseminate, distribute or copy this email Please notify
Grab Group immediately if you have received this by mistake and delete
this email from your system. Email transmission cannot be guaranteed
to be secure or error-free as any information therein could be
intercepted, corrupted, lost, destroyed, delayed or incomplete, or
contain viruses. Grab Group do not accept liability for any errors or
omissions in the contents of this email arises as a result of email
transmission. All intellectual property rights in this email and
attachments therein shall remain vested in Grab Group, unless
otherwise provided by law.
