Hey Jeremy, Thanks for kicking off this discussion. As a Flink user, I too struggled with the lack of HTTP support and rolled my own with AsyncIO. Reading through the FLIP, I just have a few general questions and comments.
* It is not clear to me if multiple HTTP methods are supported or not? It's listed in "Limitations" that only POSTs are allowed, but the constructor accepts a "method" argument. * More of a nit, the FLIP contains a lot of code, making it feel a bit more like a PR already. It would be easier to understand the proposed interfaces alone, and keep the implementation POC as a separate link IMO. Since there are so many different types of HTTP APIs, and many different ways of using them, I think the proposal would benefit from taking a more general approach to both request building and response handling. For instance, some APIs may return hints for retry that are not contained in the status code alone (e.g., a "retry-after" header or such + a 429 status code). Can we already think about how to more generally expose these two concepts? For the retries, it might be too idealistic, but standardizing on a retry interface like the one proposed in FLIP-232[1] would make these aysnc/http APIs feel much more aligned. wdyt? Austin [1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883963 On Tue, May 17, 2022 at 10:21 AM Ber, Jeremy <jd...@amazon.com.invalid> wrote: > Hi there, We would like to start a discussion thread on FLIP-233: > Introduce HTTP Connector< > https://cwiki.apache.org/confluence/display/FLINK/FLIP-233%3A+Introduce+HTTP+Connector> > where we propose to create a connector for delivering arbitrary data > packets from Apache Flink to an HTTP Endpoint. This connector will give > users flexibility to deliver data to any destination which supports REST > endpoints. This includes REST APIs, Amazon Lambda, users internal or > external consumers, among other options. > > Looking forward to your feedback. > > Thank you, > Jeremy Ber > >
