Thanks, Till! On Wed, May 30, 2018 at 10:39 AM, Till Rohrmann <trohrm...@apache.org> wrote:
> Hi Juho, > > I created a JIRA issue for the problem [1]. Yes, the solution would be to > use only GET requests. Either we wrap our requests in a GET request or we > change our handlers to accept GET requests only. One thing which we have to > fix first is that also the jar file upload goes through REST. > > [1] https://issues.apache.org/jira/browse/FLINK-9478 > > Cheers, > Till > > On Wed, May 30, 2018 at 9:07 AM, Juho Autio <juho.au...@rovio.com> wrote: > >> Hi, I tried to search Flink Jira for this but couldn't find a ticket to >> match. If there's no ticket yet, did I understand correctly though, that >> you would be open to support GET as an alternative method for all of >> Flink's REST operations? >> >> Now that 1.5 was released, it's a pity that this problem kind of prevents >> us from upgrading – or spend time on creating a workaround. >> >> >> On Thu, Apr 5, 2018 at 9:10 PM, Till Rohrmann <trohrm...@apache.org> >> wrote: >> >>> This improvement is unfortunately out of scope for the 1.5 release since >>> the feature freeze is already quite some time ago. But I hope that this >>> improvement will make it into the 1.6 release. >>> >>> Cheers, >>> Till >>> >>> On Thu, Apr 5, 2018 at 4:45 PM, Juho Autio <juho.au...@rovio.com> wrote: >>> >>>> Thanks for the answer. Wrapping with GET sounds good to me. You said >>>> next version; do you mean that Flink 1.5 would already include this >>>> improvement when it's released? >>>> >>>> On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <trohrm...@apache.org> >>>> wrote: >>>> >>>>> Hi Juho, >>>>> >>>>> you are right that due to a limitation in the Yarn proxy [1] we cannot >>>>> directly contact the cluster through the Yarn proxy. >>>>> >>>>> The way it works at the moment is that the Flink client retrieves the >>>>> AM's hostname through the ApplicationReport and then directly talks to the >>>>> AM. This of course requires that one can reach the respective Yarn >>>>> container. This is a limitation, though not a regression, which we want to >>>>> improve with the next version of Flink. An idea would be to wrap the REST >>>>> calls in a GET call to make them pass through the Yarn proxy. >>>>> >>>>> [1] https://issues.apache.org/jira/browse/YARN-2084 >>>>> >>>>> Cheers, >>>>> Till >>>>> >>>>> On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <fhue...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Juho, >>>>>> >>>>>> Thanks for raising this point! >>>>>> >>>>>> I'll add Chesnay and Till to the thread who contributed to the REST >>>>>> API. >>>>>> >>>>>> Best, Fabian >>>>>> >>>>>> 2018-04-04 15:02 GMT+02:00 Juho Autio <juho.au...@rovio.com>: >>>>>> >>>>>>> I just learned that Flink savepoints API was refactored to require >>>>>>> using HTTP POST. >>>>>>> >>>>>>> That's fine otherwise, but makes life harder when Flink is run on >>>>>>> top of YARN. >>>>>>> >>>>>>> I've added example calls below to show how POST is declined by >>>>>>> the hadoop-yarn-server-web-proxy*, which only supports GET and PUT. >>>>>>> >>>>>>> Can you think of any solution to this? If I would be able to >>>>>>> determine the actual host & port for Flink UI, I could use that instead >>>>>>> of >>>>>>> the proxy address. But that would probably require opening at least one >>>>>>> more port, so it's not the optimal solution either. Ideally I would have >>>>>>> Flink REST API completely accessible with GET and PUT methods. >>>>>>> >>>>>>> To me it seems like AWS EMR will also hit this issue as soon as they >>>>>>> start supporting Flink 1.5, because they seem to run Flink as a YARN >>>>>>> app. >>>>>>> >>>>>>> >>>>>>> *) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-p >>>>>>> roject/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web >>>>>>> -proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/ >>>>>>> WebAppProxyServlet.java#L296-L306 >>>>>>> >>>>>>> >>>>>>> $ http POST http://10.0.10.71:20888/proxy/ >>>>>>> application_1522844153347_0001/jobs/652e207f8578574d4a322e23 >>>>>>> d4f8b908/checkpoints >>>>>>> >>>>>>> HTTP/1.1 405 HTTP method POST is not supported by this URL >>>>>>> Cache-Control: must-revalidate,no-cache,no-store >>>>>>> Content-Length: 1523 >>>>>>> Content-Type: text/html; charset=iso-8859-1 >>>>>>> Date: Wed, 04 Apr 2018 12:48:05 GMT >>>>>>> Date: Wed, 04 Apr 2018 12:48:05 GMT >>>>>>> Pragma: no-cache >>>>>>> Pragma: no-cache >>>>>>> >>>>>>> <html> >>>>>>> <head> >>>>>>> <meta http-equiv="Content-Type" content="text/html; >>>>>>> charset=ISO-8859-1"/> >>>>>>> <title>Error 405 HTTP method POST is not supported by this >>>>>>> URL</title> >>>>>>> </head> >>>>>>> <body><h2>HTTP ERROR 405</h2> >>>>>>> <p>Problem accessing /proxy/application_15228441533 >>>>>>> 47_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason: >>>>>>> <pre> HTTP method POST is not supported by this URL</pre></p><hr >>>>>>> /><i><small>Powered by Jetty://</small></i><br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> <br/> >>>>>>> >>>>>>> </body> >>>>>>> </html> >>>>>>> >>>>>>> >>>>>>> $ http PUT http://10.0.10.71:20888/proxy/ >>>>>>> application_1522844153347_0001/jobs/652e207f8578574d4a322e23 >>>>>>> d4f8b908/checkpoints >>>>>>> >>>>>>> HTTP/1.1 404 Not Found >>>>>>> Access-Control-Allow-Origin: * >>>>>>> Cache-Control: no-cache >>>>>>> Content-Length: 25 >>>>>>> Content-Type: application/json; charset=UTF-8 >>>>>>> Date: Wed, 04 Apr 2018 12:48:09 GMT >>>>>>> Date: Wed, 04 Apr 2018 12:48:09 GMT >>>>>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT >>>>>>> Expires: Wed, 04 Apr 2018 12:48:09 GMT >>>>>>> Pragma: no-cache >>>>>>> Pragma: no-cache >>>>>>> >>>>>>> { >>>>>>> "errors": [ >>>>>>> "Not found." >>>>>>> ] >>>>>>> } >>>>>>> >>>>>>> ^ expected to get this from flink because there's no PUT >>>>>>> /checkpoints. >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
