If you could open a JIRA this would be great.
On 21.06.2018 09:07, sihua zhou wrote:
Hi Yow,
I had a look at the related code, I think this seems like a bug. Flink
use Checkpoint path's FileSystem to create the output stream for the
Savepoint, but in your case the checkpoint & savepoint are not using
the same file system. A workaround is to use the same file system for
both checkpoint & savepoint.
Best, Sihua
On 06/21/2018 14:07，Siew Wai Yow<[email protected]>
<mailto:[email protected]> wrote：
Thanks Chesnay, the application will take value from
"state.savepoints.dir" as default if set target-directory to nul.
But then it trying to create the directory in local machine, which
caused the below error because it is a HDFS directory. The same
URL works in previous Flink 1.3.2. Is something break in Flink
1.5.0? Or anything we need to extra configure? Thank you.
Caused by: java.io.IOException: *Mkdirs failed to create*
hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
at
org.apache.flink.core.fs.local.*LocalFileSystem*.create(LocalFileSystem.java:271)
at
org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
at
org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
... 25 more
flink-conf.yaml:
--------------------------------
state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints
-Yow
------------------------------------------------------------------------
*From:* Chesnay Schepler <[email protected]
<mailto:[email protected]>>
*Sent:* Wednesday, June 20, 2018 3:15 PM
*To:* [email protected] <mailto:[email protected]>
*Subject:* Re: Questions regarding to Flink 1.5.0 REST API change
I think you can set the target-directory to null. But I'm not sure
why this particular request requires this, other request allow
optional fields to simply be ommitted...
On 20.06.2018 06:12, Siew Wai Yow wrote:
Hi all,
Seems pass in target-directory is a must now for checkpoints REST
API, and the status will not response with save point directory
anymore. I can pass in but the information is redundant with the
same already defined in flink-config.yaml. May I know is there a
way to retrieve the save point directory from flink-config.yaml
in flink application? I am not able to get it from
env.getConfig(). Thank you.
------------------------------------------------------------------------
*From:* Chesnay Schepler <[email protected]>
<mailto:[email protected]>
*Sent:* Tuesday, June 19, 2018 11:55 PM
*To:* [email protected] <mailto:[email protected]>
*Subject:* Re: Questions regarding to Flink 1.5.0 REST API change
1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the
"mode" query parameter
2. POST to /jobs/:jobid/savepoints, with a json payload. Returns
a trigger id, used for 3).
{|
"target-directory" : { |
|"type" : "string" }, "cancel-job" : { "type" : "boolean" } } |
3. GET to /jobs/:jobid/savepoints/:triggerid
On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "*/jobs/:jobid*"
On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow
<[email protected] <mailto:[email protected]>> wrote:
Hi,
Regarding to Flink 1.5.0 REST API breaking change,
* /The REST API to cancel a job was changed./
* /The REST API to cancel a job with savepoint was changed./
I have few dump questions,
1. Any replacement for cancellation ONLY without
save-point? Only found "*/jobs/:jobid/savepoints*".
2. For "*/jobs/:jobid/savepoints*", how could I form the
URL with cancellation and with directory afterward?
1.
http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
3. Any cancellation progress monitoring in 1.5 like
previous version 1.3/1.4?
1. previous version:
/jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
Thank you.
Regards,
Yow
