Hi David, Big +1 from my side — this is great work, I went through the code and the improvements you’ve outlined clearly show the connector has matured significantly.
Also, I wanted to bring up a forward-looking idea: do you (and the community) see potential in evolving this into a broader repository for *RPC-style connectors*, including other protocols like gRPC or TChannel? At Uber, we’re actively working on consolidating various transport-layer integrations under a single umbrella, and a modular structure around a shared core (e.g., for retry, caching, auth) could help reduce duplication while keeping protocol-specific implementations cleanly separated, something like: > flink-rpc-connectors/ > ├── http-connector/ > ├── grpc-connector/ > ├── shared-utils/ (caching, retry, auth, metrics, etc.) Would love to hear thoughts from the community in this direction. Happy to contribute these sinks if they can live alongside the HTTP connector under the same modular repo ? Thanks. On Fri, May 23, 2025 at 4:42 PM David Radley <david_rad...@uk.ibm.com> wrote: > Hi, > Flip-233 [2] has been closed due to lack of capacity. I and others have > been working on enhancing the Flink getIndata HTTP connector[1]. This > connector is widely used. In the last year this connector has been > substantially improved. I want to discuss with the community whether it has > enough capabilities to donate to Flink. > > The current HTTP Connector: > > * works with SQL and datastream APIs > * manages a thread pool to help with scaling under load > * provides lookup join and sink capabilities > * ability to customize > * 90% junit coverage > > More recent enhancements include: > > * a retry mechanism ( that was asked for in Flips previous > discussions) [3] > * a new query creator, that allows json rest calls to map query param, > body, path request information from table columns; without needing to write > your own java, [4] > * standard Flink caching using the same mechanism as JDBC [5] > * supports OIDC [6] > > > > The above capabilities are useful for many scenarios. > One omission is that it is not currently a scan source. Though this is > being discussed [7]. > > Is there an appetite for the connector to be donated pretty much as-is > (with changes to make it fit with the existing Flink conventions – package > name, licenses etc). > > I am happy to drive and do this work, if the community is supportive of it. > > Kind regards, David > > > [1] > https://github.com/getindata/flink-http-connector > [2] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-233%3A+Introduce+HTTP+Connector > [3] > https://github.com/getindata/flink-http-connector/pull/148 > [4] > https://github.com/getindata/flink-http-connector/pull/149 > [5] > https://github.com/getindata/flink-http-connector/pull/94 > [6] > > https://github.com/getindata/flink-http-connector?tab=readme-ov-file#oidc-bearer-authentication > [7] > https://github.com/getindata/flink-http-connector/issues/41 > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > Winchester, Hampshire SO21 2JN >
