Hi Juho, you are right that due to a limitation in the Yarn proxy [1] we cannot directly contact the cluster through the Yarn proxy.
The way it works at the moment is that the Flink client retrieves the AM's hostname through the ApplicationReport and then directly talks to the AM. This of course requires that one can reach the respective Yarn container. This is a limitation, though not a regression, which we want to improve with the next version of Flink. An idea would be to wrap the REST calls in a GET call to make them pass through the Yarn proxy. [1] https://issues.apache.org/jira/browse/YARN-2084 Cheers, Till On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <fhue...@gmail.com> wrote: > Hi Juho, > > Thanks for raising this point! > > I'll add Chesnay and Till to the thread who contributed to the REST API. > > Best, Fabian > > 2018-04-04 15:02 GMT+02:00 Juho Autio <juho.au...@rovio.com>: > >> I just learned that Flink savepoints API was refactored to require using >> HTTP POST. >> >> That's fine otherwise, but makes life harder when Flink is run on top of >> YARN. >> >> I've added example calls below to show how POST is declined by >> the hadoop-yarn-server-web-proxy*, which only supports GET and PUT. >> >> Can you think of any solution to this? If I would be able to determine >> the actual host & port for Flink UI, I could use that instead of the proxy >> address. But that would probably require opening at least one more port, so >> it's not the optimal solution either. Ideally I would have Flink REST API >> completely accessible with GET and PUT methods. >> >> To me it seems like AWS EMR will also hit this issue as soon as they >> start supporting Flink 1.5, because they seem to run Flink as a YARN app. >> >> >> *) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-p >> roject/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server- >> web-proxy/src/main/java/org/apache/hadoop/yarn/server/ >> webproxy/WebAppProxyServlet.java#L296-L306 >> >> >> $ http POST http://10.0.10.71:20888/proxy/application_1522844153347_0001 >> /jobs/652e207f8578574d4a322e23d4f8b908/checkpoints >> >> HTTP/1.1 405 HTTP method POST is not supported by this URL >> Cache-Control: must-revalidate,no-cache,no-store >> Content-Length: 1523 >> Content-Type: text/html; charset=iso-8859-1 >> Date: Wed, 04 Apr 2018 12:48:05 GMT >> Date: Wed, 04 Apr 2018 12:48:05 GMT >> Pragma: no-cache >> Pragma: no-cache >> >> <html> >> <head> >> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> >> <title>Error 405 HTTP method POST is not supported by this URL</title> >> </head> >> <body><h2>HTTP ERROR 405</h2> >> <p>Problem accessing /proxy/application_15228441533 >> 47_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason: >> <pre> HTTP method POST is not supported by this URL</pre></p><hr >> /><i><small>Powered by Jetty://</small></i><br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> <br/> >> >> </body> >> </html> >> >> >> $ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001 >> /jobs/652e207f8578574d4a322e23d4f8b908/checkpoints >> >> HTTP/1.1 404 Not Found >> Access-Control-Allow-Origin: * >> Cache-Control: no-cache >> Content-Length: 25 >> Content-Type: application/json; charset=UTF-8 >> Date: Wed, 04 Apr 2018 12:48:09 GMT >> Date: Wed, 04 Apr 2018 12:48:09 GMT >> Expires: Wed, 04 Apr 2018 12:48:09 GMT >> Expires: Wed, 04 Apr 2018 12:48:09 GMT >> Pragma: no-cache >> Pragma: no-cache >> >> { >> "errors": [ >> "Not found." >> ] >> } >> >> ^ expected to get this from flink because there's no PUT /checkpoints. >> > >
