I'm not aware of any plans to change this behavior. Overall the
community is rather split on what role the web-submission should play;
whether it should be a genuine way to submit jobs with the same
capabilities as the CLI, just for prototyping or not exist at all.
As it stands we are somewhat hovering around the prototyping-take, so a
more appropriate long-term solution would be to allow the CLI (which
after all also goes through the REST API) to support
proxies/authentication etc., but that is likely blocked on replacing our
HTTP client with some off-the-shelf library.
One thing to note is that methods are generally not considered to be
used for production-code, and are more for prototyping.
Ideally you can find a way to simply not rely on these methods.
How/whether this can be done depends of course on the use-case; for
example let's say a job is executed and dependent on the result of
count/collect a second job should be executed. An alternative approach
might write equivalent data to some external system, and then read it
back on the client-side to orchestrate the scheduling of other jobs.
Another hacky option might be to use the CLI, but intercepting the
message and enriching it with additional authentication information?
On 1/5/2021 5:17 PM, Adam Roberts wrote:
Thanks Chesnay for the prompt response - ah, so my cunning plan to use
execution.attached=true doesn't sound so reasonable now then (I was
going to look at providing that as a programArg next).
I did find, in
https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html,
this which I assume you are referring to
/Web Submission behaves the same as detached mode./
/WithFLINK-16657
<https://issues.apache.org/jira/browse/FLINK-16657>the web submission
logic changes and it exposes the same behavior as submitting a job
through the CLI in detached mode. This implies that, for instance,
jobs based on the DataSet API that were using sinks
like|print()|,|count()|or|collect()|will now throw an exception while
before the output was simply never printed. See also comments on
relatedPR <https://github.com/apache/flink/pull/11460>./
So, here's a question - if we are advised to use a proxy to support
alternative auth mechanisms, and those mechanisms don’t work with the
CLI (thus forcing the use of curl)...how are we supposed to submit a
job with print(), count() or collect() etc?
I know you've said it's not supported, but is that an "at the moment"
kinda thing? Is this something planned or something you think I should
create a JIRA issue for?
Thanks again, much appreciated
----- Original message -----
From: Chesnay Schepler <ches...@apache.org>
To: Adam Roberts <arobe...@uk.ibm.com>, user@flink.apache.org
Cc:
Subject: [EXTERNAL] Re: Submitting a job in non-blocking mode
using curl and the REST API
Date: Tue, Jan 5, 2021 4:07 PM
All jobs going through the web-submission are run in detached mode
for technical reasons (blocking of threads, and information having
to be transported back to the JobManager for things like collect()).
You unfortunately cannot run non-detached/attached/blocking jobs
via the web submission, which includes the WordCount example
because it uses specific methods (the ones mentioned in the
exception; collect, print, printToErr, count).
In other words, your setup appears to be fine correctly, you are
just trying to do something that is not supported.
On 1/5/2021 4:07 PM, Adam Roberts wrote:
Hey everyone, I've got an awesome looking Flink cluster set up
withweb.submit.enable=true, and plenty of bash for handling jar
upload and then submission to a JobManager - all good so far.
Unfortunately, when I try to submit the classic WordCount
example, I get a massive error with the jist of it being:
/"Job was submitted in detached mode. Results of job execution,
such as accumulators, runtime, etc. are not available. Please
make sure your program doesn't call an eager execution function
[collect, print, printToErr, count]."/
*So, how do I run it *not* in detached mode using curl please?*
I'm intentionally not using the Flink CLI because I am using an
nginx with auth proxy set up - so I'm doing everything with curl,
in a bash script - (so, two requests - one to upload the jar,
then I get the ID from the response, and then submit the job with
that ID).
At
https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/rest_api.html
<https://urldefense.proofpoint.com/v2/url?u=https-3A__ci.apache.org_projects_flink_flink-2Ddocs-2Drelease-2D1.12_ops_rest-5Fapi.html&d=DwMDaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=H9t5diP_TJ2r31Y5l1LJglV9t2TPNUIeI6HOPS53w-E&m=zXEJ-FoxxvefQo1SMmWKWs3U3KOBFC5qjVw4elFBpQY&s=aznT871wIG0lvnPYEsDmhxENTV39R5sW4FVhGI833_s&e=>
if
you ctrl-f for /run, there's nothing obvious that indicates how I
can run in blocking mode - the biggest clue I've got is
`programArg`. So I'm wondering if I can provide that somehow.
For those who prefer code:
/curl ${auth_options} ${self_signed_flag} ${ca_cert_flag} -X POST
-H "Content-Type:
application/json"https://${JOBMANAGER}/jars/${uploaded_jar_string}/run$programArgsToUse
<https://urldefense.proofpoint.com/v2/url?u=https-3A__-2524-257Bjobmanager-257D_jars_-24-257Buploaded-5Fjar-5Fstring-257D_run-24programArgsToUse&d=DwMDaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=H9t5diP_TJ2r31Y5l1LJglV9t2TPNUIeI6HOPS53w-E&m=zXEJ-FoxxvefQo1SMmWKWs3U3KOBFC5qjVw4elFBpQY&s=V8EjsrlypfkPrhudGgdqy9COq9h8tLi546dzGVW_26k&e=>/
Whereby/programArgsToUse/is user args, and I'm cool with them
being query parameters for now - I think.
I'm passing them on the end with:
if [[ ! -z $program_args ]] ; then
programArgsToUse="?programArg=$program_args"
fi
so my eventual curl looks like this. But, I'm really just
guessing what the detached argument is...
curl --cacert /etc/ssl/tester/certs/ca.crt -X POST -H
'Content-Type: application/json'
'https://tester-minimal-tls-sample-jobmanager:8081/jars/fdc7684f-323d-49fa-a60a-96683d953be8_WordCount.jar/run?programArg=detached=false
<https://urldefense.proofpoint.com/v2/url?u=https-3A__tester-2Dminimal-2Dtls-2Dsample-2Djobmanager-3A8081_jars_fdc7684f-2D323d-2D49fa-2Da60a-2D96683d953be8-5FWordCount.jar_run-3FprogramArg-3Ddetached-3Dfalse&d=DwMDaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=H9t5diP_TJ2r31Y5l1LJglV9t2TPNUIeI6HOPS53w-E&m=zXEJ-FoxxvefQo1SMmWKWs3U3KOBFC5qjVw4elFBpQY&s=N9D8drVPEORUAZsV8On7JbbtYJlxuiUXnZuUcG0YEDI&e=>'
(obviously, what's at the end looks really wrong, but IDK what to
use)
The only mention of "detach" I see documented is at
https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html
<https://urldefense.proofpoint.com/v2/url?u=https-3A__ci.apache.org_projects_flink_flink-2Ddocs-2Dstable_deployment_config.html&d=DwMDaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=H9t5diP_TJ2r31Y5l1LJglV9t2TPNUIeI6HOPS53w-E&m=zXEJ-FoxxvefQo1SMmWKWs3U3KOBFC5qjVw4elFBpQY&s=SnEahFltZn_ajLk_2KvZ4IZf6Rl39_mf4Ug8zgWStCw&e=>,
and that says by default it's not in detached mode for execution.
If there are any better docs or examples, please send them my way
- or if you've spotted me being just plain silly with my
bash, that would be fantastic to point out.
Thanks a lot in advance, cheers, happy to share any more
code/background if need be
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with
number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth,
Hampshire PO6 3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with
number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
