Hey Hao Sun, this has been changed recently [1] in order to properly support failover in job cluster mode.
A workaround for you would be to add an application identifier to the checkpoint path of each application, resulting in S3 paths like application-XXXX/00...00/chk-64. Is that a feasible solution? As a side note: It was considered to keep the job ID fixed, but make it configurable (e.g. by providing a --job-id argument) which would also help to avoid this situation, but I'm not aware of any concrete plans to move forward with that approach. Best, Ufuk [1] https://issues.apache.org/jira/projects/FLINK/issues/FLINK-10291 On Sun, Nov 4, 2018 at 3:39 AM Hao Sun <ha...@zendesk.com> wrote: > > I am wondering if I can customize job_id for job cluster mode. Currently it > is always 00000000000000000000000000000000. I am running multiple job > clusters and sharing s3, it means checkpoints will be shared by different > jobs as well e.g. 00000000000000000000000000000000/chk-64, how can I avoid > this? > > Thanks
