Thanks for the answer. Wrapping with GET sounds good to me. You said next version; do you mean that Flink 1.5 would already include this improvement when it's released?
On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <trohrm...@apache.org> wrote: > Hi Juho, > > you are right that due to a limitation in the Yarn proxy [1] we cannot > directly contact the cluster through the Yarn proxy. > > The way it works at the moment is that the Flink client retrieves the AM's > hostname through the ApplicationReport and then directly talks to the AM. > This of course requires that one can reach the respective Yarn container. > This is a limitation, though not a regression, which we want to improve > with the next version of Flink. An idea would be to wrap the REST calls in > a GET call to make them pass through the Yarn proxy. > > [1] https://issues.apache.org/jira/browse/YARN-2084 > > Cheers, > Till > > On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <fhue...@gmail.com> wrote: > >> Hi Juho, >> >> Thanks for raising this point! >> >> I'll add Chesnay and Till to the thread who contributed to the REST API. >> >> Best, Fabian >> >> 2018-04-04 15:02 GMT+02:00 Juho Autio <juho.au...@rovio.com>: >> >>> I just learned that Flink savepoints API was refactored to require using >>> HTTP POST. >>> >>> That's fine otherwise, but makes life harder when Flink is run on top of >>> YARN. >>> >>> I've added example calls below to show how POST is declined by >>> the hadoop-yarn-server-web-proxy*, which only supports GET and PUT. >>> >>> Can you think of any solution to this? If I would be able to determine >>> the actual host & port for Flink UI, I could use that instead of the proxy >>> address. But that would probably require opening at least one more port, so >>> it's not the optimal solution either. Ideally I would have Flink REST API >>> completely accessible with GET and PUT methods. >>> >>> To me it seems like AWS EMR will also hit this issue as soon as they >>> start supporting Flink 1.5, because they seem to run Flink as a YARN app. >>> >>> >>> *) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-p >>> roject/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web >>> -proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/ >>> WebAppProxyServlet.java#L296-L306 >>> >>> >>> $ http POST http://10.0.10.71:20888/proxy/application_1522844153347_0001 >>> /jobs/652e207f8578574d4a322e23d4f8b908/checkpoints >>> >>> HTTP/1.1 405 HTTP method POST is not supported by this URL >>> Cache-Control: must-revalidate,no-cache,no-store >>> Content-Length: 1523 >>> Content-Type: text/html; charset=iso-8859-1 >>> Date: Wed, 04 Apr 2018 12:48:05 GMT >>> Date: Wed, 04 Apr 2018 12:48:05 GMT >>> Pragma: no-cache >>> Pragma: no-cache >>> >>> <html> >>> <head> >>> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> >>> <title>Error 405 HTTP method POST is not supported by this URL</title> >>> </head> >>> <body><h2>HTTP ERROR 405</h2> >>> <p>Problem accessing /proxy/application_15228441533 >>> 47_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason: >>> <pre> HTTP method POST is not supported by this URL</pre></p><hr >>> /><i><small>Powered by Jetty://</small></i><br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> <br/> >>> >>> </body> >>> </html> >>> >>> >>> $ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001 >>> /jobs/652e207f8578574d4a322e23d4f8b908/checkpoints >>> >>> HTTP/1.1 404 Not Found >>> Access-Control-Allow-Origin: * >>> Cache-Control: no-cache >>> Content-Length: 25 >>> Content-Type: application/json; charset=UTF-8 >>> Date: Wed, 04 Apr 2018 12:48:09 GMT >>> Date: Wed, 04 Apr 2018 12:48:09 GMT >>> Expires: Wed, 04 Apr 2018 12:48:09 GMT >>> Expires: Wed, 04 Apr 2018 12:48:09 GMT >>> Pragma: no-cache >>> Pragma: no-cache >>> >>> { >>> "errors": [ >>> "Not found." >>> ] >>> } >>> >>> ^ expected to get this from flink because there's no PUT /checkpoints. >>> >> >> >
