That's quite weird that it tries to us the local file-system. Maybe it
derives the FS from state.backend.fs.checckpointdir, but uses it for
savepoints.dir?
What happens if you set state.backend.fs.checkpointdir also to HDFS?
On 21.06.2018 08:07, Siew Wai Yow wrote:
Thanks Chesnay, the application will take value from
"state.savepoints.dir" as default if set target-directory to nul. But
then it trying to create the directory in local machine, which
caused the below error because it is a HDFS directory. The same URL
works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or
anything we need to extra configure? Thank you.
Caused by: java.io.IOException: *Mkdirs failed to create*
hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
at
org.apache.flink.core.fs.local.*LocalFileSystem*.create(LocalFileSystem.java:271)
at
org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
at
org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
... 25 more
flink-conf.yaml:
--------------------------------
state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints
-Yow
------------------------------------------------------------------------
*From:* Chesnay Schepler <[email protected]>
*Sent:* Wednesday, June 20, 2018 3:15 PM
*To:* [email protected]
*Subject:* Re: Questions regarding to Flink 1.5.0 REST API change
I think you can set the target-directory to null. But I'm not sure why
this particular request requires this, other request allow optional
fields to simply be ommitted...
On 20.06.2018 06:12, Siew Wai Yow wrote:
Hi all,
Seems pass in target-directory is a must now for checkpoints REST
API, and the status will not response with save point directory
anymore. I can pass in but the information is redundant with the same
already defined in flink-config.yaml. May I know is there a way to
retrieve the save point directory from flink-config.yaml in flink
application? I am not able to get it from env.getConfig(). Thank you.
------------------------------------------------------------------------
*From:* Chesnay Schepler <[email protected]> <mailto:[email protected]>
*Sent:* Tuesday, June 19, 2018 11:55 PM
*To:* [email protected] <mailto:[email protected]>
*Subject:* Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode"
query parameter
2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a
trigger id, used for 3).
{|
"target-directory" : { |
|"type" : "string" }, "cancel-job" : { "type" : "boolean" } } |
3. GET to /jobs/:jobid/savepoints/:triggerid
On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "*/jobs/:jobid*"
On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <[email protected]
<mailto:[email protected]>> wrote:
Hi,
Regarding to Flink 1.5.0 REST API breaking change,
* /The REST API to cancel a job was changed./
* /The REST API to cancel a job with savepoint was changed./
I have few dump questions,
1. Any replacement for cancellation ONLY without save-point?
Only found "*/jobs/:jobid/savepoints*".
2. For "*/jobs/:jobid/savepoints*", how could I form the URL
with cancellation and with directory afterward?
1. http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
3. Any cancellation progress monitoring in 1.5 like previous
version 1.3/1.4?
1. previous version:
/jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
Thank you.
Regards,
Yow
