This improvement is unfortunately out of scope for the 1.5 release since the feature freeze is already quite some time ago. But I hope that this improvement will make it into the 1.6 release.
Cheers, Till On Thu, Apr 5, 2018 at 4:45 PM, Juho Autio <juho.au...@rovio.com> wrote: > Thanks for the answer. Wrapping with GET sounds good to me. You said next > version; do you mean that Flink 1.5 would already include this improvement > when it's released? > > On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <trohrm...@apache.org> > wrote: > >> Hi Juho, >> >> you are right that due to a limitation in the Yarn proxy [1] we cannot >> directly contact the cluster through the Yarn proxy. >> >> The way it works at the moment is that the Flink client retrieves the >> AM's hostname through the ApplicationReport and then directly talks to the >> AM. This of course requires that one can reach the respective Yarn >> container. This is a limitation, though not a regression, which we want to >> improve with the next version of Flink. An idea would be to wrap the REST >> calls in a GET call to make them pass through the Yarn proxy. >> >> [1] https://issues.apache.org/jira/browse/YARN-2084 >> >> Cheers, >> Till >> >> On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <fhue...@gmail.com> wrote: >> >>> Hi Juho, >>> >>> Thanks for raising this point! >>> >>> I'll add Chesnay and Till to the thread who contributed to the REST API. >>> >>> Best, Fabian >>> >>> 2018-04-04 15:02 GMT+02:00 Juho Autio <juho.au...@rovio.com>: >>> >>>> I just learned that Flink savepoints API was refactored to require >>>> using HTTP POST. >>>> >>>> That's fine otherwise, but makes life harder when Flink is run on top >>>> of YARN. >>>> >>>> I've added example calls below to show how POST is declined by >>>> the hadoop-yarn-server-web-proxy*, which only supports GET and PUT. >>>> >>>> Can you think of any solution to this? If I would be able to determine >>>> the actual host & port for Flink UI, I could use that instead of the proxy >>>> address. But that would probably require opening at least one more port, so >>>> it's not the optimal solution either. Ideally I would have Flink REST API >>>> completely accessible with GET and PUT methods. >>>> >>>> To me it seems like AWS EMR will also hit this issue as soon as they >>>> start supporting Flink 1.5, because they seem to run Flink as a YARN app. >>>> >>>> >>>> *) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-p >>>> roject/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web >>>> -proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/ >>>> WebAppProxyServlet.java#L296-L306 >>>> >>>> >>>> $ http POST http://10.0.10.71:20888/proxy/ >>>> application_1522844153347_0001/jobs/652e207f8578574d4a322e23 >>>> d4f8b908/checkpoints >>>> >>>> HTTP/1.1 405 HTTP method POST is not supported by this URL >>>> Cache-Control: must-revalidate,no-cache,no-store >>>> Content-Length: 1523 >>>> Content-Type: text/html; charset=iso-8859-1 >>>> Date: Wed, 04 Apr 2018 12:48:05 GMT >>>> Date: Wed, 04 Apr 2018 12:48:05 GMT >>>> Pragma: no-cache >>>> Pragma: no-cache >>>> >>>> <html> >>>> <head> >>>> <meta http-equiv="Content-Type" content="text/html; >>>> charset=ISO-8859-1"/> >>>> <title>Error 405 HTTP method POST is not supported by this URL</title> >>>> </head> >>>> <body><h2>HTTP ERROR 405</h2> >>>> <p>Problem accessing /proxy/application_15228441533 >>>> 47_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason: >>>> <pre> HTTP method POST is not supported by this URL</pre></p><hr >>>> /><i><small>Powered by Jetty://</small></i><br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> <br/> >>>> >>>> </body> >>>> </html> >>>> >>>> >>>> $ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001 >>>> /jobs/652e207f8578574d4a322e23d4f8b908/checkpoints >>>> >>>> HTTP/1.1 404 Not Found >>>> Access-Control-Allow-Origin: * >>>> Cache-Control: no-cache >>>> Content-Length: 25 >>>> Content-Type: application/json; charset=UTF-8 >>>> Date: Wed, 04 Apr 2018 12:48:09 GMT >>>> Date: Wed, 04 Apr 2018 12:48:09 GMT >>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT >>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT >>>> Pragma: no-cache >>>> Pragma: no-cache >>>> >>>> { >>>> "errors": [ >>>> "Not found." >>>> ] >>>> } >>>> >>>> ^ expected to get this from flink because there's no PUT /checkpoints. >>>> >>> >>> >> >
