Hi Rainie,
if I remember correctly (unfortunately I don't have a S3 deployment at hand
to try it out), then in v1.9 you should find the data files for the
checkpoint under s3a://{bucket
name}/dev/checkpoints/_entropy_/{job_id}/chk-2230. A checkpoint consists of
these data files and a metadata file which links the individual data files
from the different operators together to a checkpoint. The metadata file
should be stored under s3a://{bucket
name}/dev/checkpoints/{job_id}/chk-2230 so that it is easily discoverable.
If the data files are also contained in s3a://{bucket
name}/dev/checkpoints/{job_id}/chk-2230, then there is some problem or the
system did not properly use the entropy functionality.
My suspicion is that with FLINK-5763 (this has been introduced with Flink
1.11) we moved the metadata file also under the entropy folder to make the
checkpoints/savepoints self-contained and relocatable.
Cheers,
Till
On Wed, Mar 17, 2021 at 10:14 PM Rainie Li <raini...@pinterest.com.invalid>
wrote:
> Thanks for checking, Till.
>
> I have a follow up question for #2, do you know why the same job cannot
> show up at the entropy checkpoint in Version 1.9.
> For example:
> *When it's running in v1.11, checkpoint path is: *
> s3a://{bucket name}/dev/checkpoints/_entropy_/{job_id}/chk-1537
> *When it's running in v1.9, checkpoint path is: *
> s3a://{bucket name}/dev/checkpoints/{job_id}/chk-2230
>
> Not sure which caused this inconsistency issue.
> Thanks
> Best regards
> Rainie
>
> On Wed, Mar 17, 2021 at 6:38 AM Till Rohrmann <trohrm...@apache.org>
> wrote:
>
> > Hi Rainie,
> >
> > 1. I think what you need to do is to look for the {job_id} in all the
> > possible sub folders of the dev/checkpoints/ folder or you extract the
> > entropy from the logs.
> >
> > 2. According to [1] entropy should only be used for the data files and
> not
> > for the metadata files. The idea was to keep the metadata path entropy
> free
> > in order to make it more easily discoverable. I can imagine that this
> > changed with FLINK-5763 [2] which was added in Flink 1.11. This
> effectively
> > means that in order to make checkpoints/savepoints self contained we
> needed
> > to add the entropy also to the metadata file paths. Moreover, this also
> > means that the entropy injection works for 1.9 and 1.11. I think it was
> > introduced with Flink 1.6.2, 1.7.0 [3].
> >
> > [1]
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-stable/deployment/filesystems/s3.html#entropy-injection-for-s3-file-systems
> > [2] https://issues.apache.org/jira/browse/FLINK-5763
> > [3] https://issues.apache.org/jira/browse/FLINK-9061
> >
> > Cheers,
> > Till
> >
> > On Tue, Mar 16, 2021 at 7:03 PM Rainie Li <raini...@pinterest.com
> .invalid>
> > wrote:
> >
> > > Hi Flink Developers.
> > >
> > > We enabled entropy injection for s3, here is our setting on Yarn
> Cluster.
> > > s3.entropy.key: _entropy_
> > > s3.entropy.length: 1
> > > state.checkpoints.dir: 's3a://{bucket name}/dev/checkpoints/_entropy_'
> > >
> > > I have two questions:
> > > 1. After enabling entropy, job's checkpoint path changed to:
> > > *s3://{bucket name}/dev/checkpoints/_entropy_/{job_id}chk-607*
> > > SInce we don't know which key is mapped to _entropy_
> > > It cannot be used to relaunch flink jobs by running
> > > *flink run -s **s3://{bucket
> > > name}/dev/checkpoints/_entropy_/{job_id}chk-607*
> > > If you also enabled entropy injection for s3, any suggestion how to
> > recover
> > > failed jobs using entropy checkpoints?
> > >
> > > 2.We added entropy settings on the Yarn cluster.
> > > But we can only see flink jobs in version 1.11 shows the entropy
> > checkpoint
> > > path.
> > > For flink jobs version 1.9, they are still using checkpoint paths
> without
> > > entropy like:
> > > *s3://{bucket name}/dev/checkpoints/{job_id}/chk-607*
> > > Is this path equal to s3://*{bucket name}*
> > > */dev/checkpoints/_entropy_/{job_id}**chk-607?*
> > > Does entropy work for v1.9? If so, why does v1.9 job show checkpoint
> > paths
> > > *without* entropy?
> > >
> > > Appreciated any suggestions.
> > > Thanks
> > > Best regards
> > > Rainie
> > >
> >
>
