Hi all, Krzysztof Zarzycki, CTO of Getindata (soon to be Xebia) here, I lead the group that developed the connector and collaborates with David on it further. Thank you David for starting the thread, as we chatted.
We as Getindata would be delighted to offer the repository to the Flink community. It serves us well on our customers' projects, but I believe it can be much more useful when broader community improves it. We would feel honored to know that this code is used by a larger group of companies, including with well known names :-) . We will still happily work on it further - as Xebia from now on. Regarding creating a repository of RPC connectors. We actually discussed within the community to create generally a single place for connectors, not limited to RPC only. This could help with discoverability of the connectors, maintain one standard among them, including its compatibility with Flink versions. What do you think about this? Thanks, Krzysztof On Thu, May 29, 2025 at 6:36 PM Sharath <dsaishar...@gmail.com> wrote: > +1 (non-binding). I think this would be a great addition as well! > > On Thu, May 29, 2025 at 9:28 AM Tom Cooper <c...@tomcooper.dev> wrote: > > > +1 (non-binding) from me! This seems like it would be a really useful > > functionality to have within Flink. > > > > Can we get some official notice (email on this thread, Github > > discussion/issue) from the GetInData folks and other maintainers that > they > > are happy to donate the project to Flink and would be willing to support > it > > once it moves? > > > > Also does this require a new FLIP or can we resurrect the original > > FLIP-233? > > > > Tom Cooper > > @tomcooper.dev | https://tomcooper.dev > > > > > > On Friday, 23 May 2025 at 12:13, David Radley <david_rad...@uk.ibm.com> > > wrote: > > > > > Hi, > > > Flip-233 [2] has been closed due to lack of capacity. I and others have > > been working on enhancing the Flink getIndata HTTP connector[1]. This > > connector is widely used. In the last year this connector has been > > substantially improved. I want to discuss with the community whether it > has > > enough capabilities to donate to Flink. > > > > > > The current HTTP Connector: > > > > > > * works with SQL and datastream APIs > > > * manages a thread pool to help with scaling under load > > > * provides lookup join and sink capabilities > > > * ability to customize > > > * 90% junit coverage > > > > > > More recent enhancements include: > > > > > > * a retry mechanism ( that was asked for in Flips previous discussions) > > [3] > > > * a new query creator, that allows json rest calls to map query param, > > body, path request information from table columns; without needing to > write > > your own java, [4] > > > * standard Flink caching using the same mechanism as JDBC [5] > > > * supports OIDC [6] > > > > > > > > > > > > The above capabilities are useful for many scenarios. > > > One omission is that it is not currently a scan source. Though this is > > being discussed [7]. > > > > > > Is there an appetite for the connector to be donated pretty much as-is > > (with changes to make it fit with the existing Flink conventions – > package > > name, licenses etc). > > > > > > I am happy to drive and do this work, if the community is supportive of > > it. > > > > > > Kind regards, David > > > > > > > > > [1] > > > https://github.com/getindata/flink-http-connector > > > [2] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-233%3A+Introduce+HTTP+Connector > > > [3] > > > https://github.com/getindata/flink-http-connector/pull/148 > > > [4] > > > https://github.com/getindata/flink-http-connector/pull/149 > > > [5] > > > https://github.com/getindata/flink-http-connector/pull/94 > > > [6] > > > > > > https://github.com/getindata/flink-http-connector?tab=readme-ov-file#oidc-bearer-authentication > > > [7] > > > https://github.com/getindata/flink-http-connector/issues/41 > > > > > > Unless otherwise stated above: > > > > > > IBM United Kingdom Limited > > > Registered in England and Wales with number 741598 > > > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > > Winchester, Hampshire SO21 2JN > > >
