Hi, The Flink http connector repository [1] has now been populated; thanks @Robert Metzger<mailto:[email protected]> @Ferenc Csaky<mailto:[email protected]> and Grzegorz for your help
Currently the connector is built against Flink 1.20, it has the code in the main branch. I would like to start the conversation about releasing it. There are some changes that we would like to do before the first release. They are 1. Flink-38480 [2] this is a port of a feature that allows the Flink job to continue after a http error and surfaces http information in metadata columns. This allows error processing to stay in the streaming environment. This approach was discussed with @Arvid Heise<mailto:[email protected]> on the dev list, a more pervasive Flinch framework change was suggested by Arvid to solve this problem generically, but he was happy we proceeded with the metadata column approach. This is merged into the repo 2. FLINK-38481 [3] A change to remove headers and response values from the logging. This is in review. There is a 3rd change [4] that may or may not make it into the first release. Because this connector is only a lookup connector it only gets the lookup keys not the filters ( which would come if the connector implemented scan). This means that if the planner optimizes the select to not be an equality then the lookup fails. The idea is we would initially enable a minimal amount of scan connector capability to allow the lookups to proceed, then look to enhance. After we have these in main. I propose we cut : * A version 1.0 based on Flink 1.20 * A version 2.0 based on Flink 2.0 * It would be useful if the HTTP connector component could be added to Jira, I do not think I have authority to add it. I am happy to do the work for these releases, but would need assistance from a committer for everything that needs write access. Please let me know your thoughts and whether you will support this approach, Kind regards, David. [1] https://github.com/apache/flink-connector-http [2] https://issues.apache.org/jira/browse/FLINK-38480 [3] https://issues.apache.org/jira/browse/FLINK-38481 [4] https://github.com/getindata/flink-http-connector/issues/41 Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
