Thanks @SihuaZhou, you are right that this is a bug. Just check the source code too. @Chesnay Schepler, <mailto:ches...@apache.org> Tested with both checkpoint and savepoint at same File system and it is working as expected.
Thanks guys! -Yow ________________________________ From: Chesnay Schepler <ches...@apache.org> Sent: Thursday, June 21, 2018 3:36 PM To: user@flink.apache.org Subject: Re: Questions regarding to Flink 1.5.0 REST API change If you could open a JIRA this would be great. On 21.06.2018 09:07, sihua zhou wrote: Hi Yow, I had a look at the related code, I think this seems like a bug. Flink use Checkpoint path's FileSystem to create the output stream for the Savepoint, but in your case the checkpoint & savepoint are not using the same file system. A workaround is to use the same file system for both checkpoint & savepoint. Best, Sihua On 06/21/2018 14:07，Siew Wai Yow<wai_...@hotmail.com><mailto:wai_...@hotmail.com> wrote： Thanks Chesnay, the application will take value from "state.savepoints.dir" as default if set target-directory to nul. But then it trying to create the directory in local machine, which caused the below error because it is a HDFS directory. The same URL works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or anything we need to extra configure? Thank you. Caused by: java.io.IOException: Mkdirs failed to create hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:271) at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121) at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348) ... 25 more flink-conf.yaml: -------------------------------- state.backend: rocksdb state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example state.backend.incremental: true state.backend.async: true state.checkpoints.num-retained: 5 state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints -Yow ________________________________ From: Chesnay Schepler <ches...@apache.org<mailto:ches...@apache.org>> Sent: Wednesday, June 20, 2018 3:15 PM To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Questions regarding to Flink 1.5.0 REST API change I think you can set the target-directory to null. But I'm not sure why this particular request requires this, other request allow optional fields to simply be ommitted... On 20.06.2018 06:12, Siew Wai Yow wrote: Hi all, Seems pass in target-directory is a must now for checkpoints REST API, and the status will not response with save point directory anymore. I can pass in but the information is redundant with the same already defined in flink-config.yaml. May I know is there a way to retrieve the save point directory from flink-config.yaml in flink application? I am not able to get it from env.getConfig(). Thank you. ________________________________ From: Chesnay Schepler <ches...@apache.org><mailto:ches...@apache.org> Sent: Tuesday, June 19, 2018 11:55 PM To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Questions regarding to Flink 1.5.0 REST API change 1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter 2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3). { "target-directory" : { "type" : "string" }, "cancel-job" : { "type" : "boolean" } } 3. GET to /jobs/:jobid/savepoints/:triggerid On 19.06.2018 17:40, Esteban Serrano wrote: For #1, you need to use a PATCH request to "/jobs/:jobid" On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wai_...@hotmail.com<mailto:wai_...@hotmail.com>> wrote: Hi, Regarding to Flink 1.5.0 REST API breaking change, * The REST API to cancel a job was changed. * The REST API to cancel a job with savepoint was changed. I have few dump questions, 1. Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints". 2. For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory afterward? * http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true?? 3. Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4? * previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId Thank you. Regards, Yow
