Hi all, Thanks for opening a discussion and/or creating a FLIP. I do think that a HTTP connector could be a powerful connector for users.
In order to get a proper HTTP connector in as part of the Apache Flink project, we'll need to have FLIP and a successful vote. When that's done, one of the PMC members can create a separate repository (something like apache/flink-connector-http). A couple of first thoughts from my end: * I think it's fine to start with either a source or a sink. Depending on how it's scoped, we either include it in one FLIP. That doesn't mean that all the work needs to be done in one go as well, but it just requires you to only have one vote. If we want to split it up (first do a sink and add a source later), we most likely need a second FLIP to cover the part that will be added later * I do think that when a source/sink is added, we can't scope it to users of either the DataStream or Table/SQL. We should make sure that we don't introduce additional sparsity. Then again, given that the connector won't be bound to Flink's release cycle (because it's hosted in a different repository), you can add additional capabilities more quickly so this might not be a really big problem. * Whenever we add a source, it would be quite nice to cover both Bounded and Unbounded sources. Most likely the source would be a Bounded source (you're making a http request, perhaps with some paginated result and that's it). However, there are quite some APIs out there that can support technologies like server sent events, long polling or websockets. That would be really nice to have as way to get to use the connector for unbounded sources. Really looking forward to moving this forward! Best regards, Martijn Op do 2 jun. 2022 om 16:54 schreef Ber, Jeremy <jd...@amazon.com.invalid>: > Hi Xuyang, > > Great feedback on the early days of this FLIP! > > With respect to the Source question, definitely want to add a source > connector in collaboration with the getindata folks below in this thread. > We will collaborate to make HTTP source changes. > > I agree that FLIP-171 is the cornerstone of this idea and this was the > original inspiration for building out this connector. As we welcome > additional collaborators the additional functionality will come, including > source connector, additional request builder utilities and Table / SQL API. > > I will add a simple example once the connector interface is fully realized. > > Thank you so much! > > - Jeremy > > On 6/1/22, 9:30 PM, "Xuyang" <xyzhong...@163.com> wrote: > > CAUTION: This email originated from outside of the organization. Do > not click links or open attachments unless you can confirm the sender and > know the content is safe. > > > > Hi, Jeremy, thanks for driving this work. I have noticed that there > have been a lot of users that need a official HTTP connector from > user-email. > I have some comments about this Flip. > > > * The title is "Introduce HTTP Connector", but it seems that this FLIP > only contains the content of HTTP sink. Is your purpose only about sink and > does this flip not involve HTTPSource? > I think you can change the title or add some descriptions about > HTTPSource. What do you think? > > > * I notice that your only attach the FLIP-171 here and left only one > method as HTTPSink function. > I think it lacks some logic about how to build a Request or deal with > the Response just Austin says. That seems that the FLIP-171 is only the > cornerstone of this flip, and the more extra changes about design different > with FLIP-171 should be reflected here. > > > * Because there will be a long way to go about this Flip, I aggree > that in the initial version we can support HTTP Sink in DataStream Api only > ; ) > > > * IMO, a simple example added in the end of the flip is necessary. > > > WDYT? > > > > -- > > Best！ > Xuyang > > > > > > 在 2022-06-01 23:32:50，"Krzysztof Chmielewski" < > krzysiek.chmielew...@gmail.com> 写道： > >Hello Jeremy, > >I'm the original author and currently main contributor to our HTTP > >connector [1] that Maciej wrote about. > > > >I'm more than happy to help and get this going. > >Should you have any questions about our implementation feel free to > contact > >me. > > > >[1]: https://github.com/getindata/flink-http-connector > > > > > >Best Regards, > >Krzysztof Chmielewski > > > > > > > >On 2022/06/01 06:38:55 Maciej Bryński wrote: > > > >> Hi Jeremy, > > > >> We as GetinData already started some work related with HTTP > connector. > > > >> As a first step we created HTTP Source [1] described in blog post > here > >[2]. > > > >> During next 3 months we are planning to create HTTP Sink based on > Flink > >1.15 new Async Sink API. > > > >> How can we help with making this sink “official” ? > > > >> > > > >> Regards, > > > >> Maciek > > > >> > > > >> [1]: > > > >> https://github.com/getindata/flink-http-connector > > > >> [2]: > > > >> > > > https://getindata.com/blog/data-enrichment-flink-sql-http-connector-flink-sql-part-one/ > > > >> > > > >> On 2022/05/17 14:21:31 "Ber, Jeremy" wrote: > > > >> > Hi there, We would like to start a discussion thread on FLIP-233: > >Introduce HTTP Connector< > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-233%3A+Introduce+HTTP+Connector > > > >where we propose to create a connector for delivering arbitrary data > >packets from Apache Flink to an HTTP Endpoint. This connector will > give > >users flexibility to deliver data to any destination which supports > REST > >endpoints. This includes REST APIs, Amazon Lambda, users internal or > >external consumers, among other options. > > > >> > > > > >> > Looking forward to your feedback. > > > >> > > > > >> > Thank you, > > > >> > Jeremy Ber > > > >> > > > > >> > > > > >Wysłane z aplikacji Poczta < > https://go.microsoft.com/fwlink/?LinkId=550986> > >dla systemu Windows > >
