+1 (non-binding) from me! This seems like it would be a really useful functionality to have within Flink.
Can we get some official notice (email on this thread, Github discussion/issue) from the GetInData folks and other maintainers that they are happy to donate the project to Flink and would be willing to support it once it moves? Also does this require a new FLIP or can we resurrect the original FLIP-233? Tom Cooper @tomcooper.dev | https://tomcooper.dev On Friday, 23 May 2025 at 12:13, David Radley <david_rad...@uk.ibm.com> wrote: > Hi, > Flip-233 [2] has been closed due to lack of capacity. I and others have been > working on enhancing the Flink getIndata HTTP connector[1]. This connector is > widely used. In the last year this connector has been substantially improved. > I want to discuss with the community whether it has enough capabilities to > donate to Flink. > > The current HTTP Connector: > > * works with SQL and datastream APIs > * manages a thread pool to help with scaling under load > * provides lookup join and sink capabilities > * ability to customize > * 90% junit coverage > > More recent enhancements include: > > * a retry mechanism ( that was asked for in Flips previous discussions) [3] > * a new query creator, that allows json rest calls to map query param, body, > path request information from table columns; without needing to write your > own java, [4] > * standard Flink caching using the same mechanism as JDBC [5] > * supports OIDC [6] > > > > The above capabilities are useful for many scenarios. > One omission is that it is not currently a scan source. Though this is being > discussed [7]. > > Is there an appetite for the connector to be donated pretty much as-is (with > changes to make it fit with the existing Flink conventions – package name, > licenses etc). > > I am happy to drive and do this work, if the community is supportive of it. > > Kind regards, David > > > [1] > https://github.com/getindata/flink-http-connector > [2] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-233%3A+Introduce+HTTP+Connector > [3] > https://github.com/getindata/flink-http-connector/pull/148 > [4] > https://github.com/getindata/flink-http-connector/pull/149 > [5] > https://github.com/getindata/flink-http-connector/pull/94 > [6] > https://github.com/getindata/flink-http-connector?tab=readme-ov-file#oidc-bearer-authentication > [7] > https://github.com/getindata/flink-http-connector/issues/41 > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > Winchester, Hampshire SO21 2JN
