Hi, Flip-233 [2] has been closed due to lack of capacity. I and others have been working on enhancing the Flink getIndata HTTP connector[1]. This connector is widely used. In the last year this connector has been substantially improved. I want to discuss with the community whether it has enough capabilities to donate to Flink.
The current HTTP Connector: * works with SQL and datastream APIs * manages a thread pool to help with scaling under load * provides lookup join and sink capabilities * ability to customize * 90% junit coverage More recent enhancements include: * a retry mechanism ( that was asked for in Flips previous discussions) [3] * a new query creator, that allows json rest calls to map query param, body, path request information from table columns; without needing to write your own java, [4] * standard Flink caching using the same mechanism as JDBC [5] * supports OIDC [6] The above capabilities are useful for many scenarios. One omission is that it is not currently a scan source. Though this is being discussed [7]. Is there an appetite for the connector to be donated pretty much as-is (with changes to make it fit with the existing Flink conventions – package name, licenses etc). I am happy to drive and do this work, if the community is supportive of it. Kind regards, David [1] https://github.com/getindata/flink-http-connector [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-233%3A+Introduce+HTTP+Connector [3] https://github.com/getindata/flink-http-connector/pull/148 [4] https://github.com/getindata/flink-http-connector/pull/149 [5] https://github.com/getindata/flink-http-connector/pull/94 [6] https://github.com/getindata/flink-http-connector?tab=readme-ov-file#oidc-bearer-authentication [7] https://github.com/getindata/flink-http-connector/issues/41 Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
