Ok, that's a bit bigger than I thought.
On Tue, Aug 30, 2022 at 10:03 AM Martijn Visser <martijnvis...@apache.org> wrote: > Hi Niels, > > With regards to ES support, we should follow the supported version that ES > supports [1]. So if we add support for ES8, we drop support for ES6. > > With regards to the used clients, we can't upgrade to the latest version of > `RestHighLevelClient`. That's because Elasticsearch has changed the license > from version 7.11 onwards to the Apache-incompatible SSPL license [2]. What > needs to happen is that the ES connector is refactored to use the new > Elasticsearch Java Client [3] which is still available under Apache 2.0 > license in order to add support for ES8 clusters. > > For now, we need to make Elasticsearch changes to both the Flink main repo > and the external one. Hopefully we can drop the ES connector from Flink's > repo with Flink 1.17. > > Best regards, > > Martijn > > [1] https://www.elastic.co/support/eol > [2] https://www.elastic.co/pricing/faq/licensing > [3] https://github.com/elastic/elasticsearch-java > > Op do 11 aug. 2022 om 16:45 schreef Niels Basjes <ni...@basjes.nl>: > > > Hi, > > > > I was building something that puts data in ES and found that the current > > connector is not compatible with an ES 8 cluster. > > > > I found that > > - The currently employed `RestHighLevelClient` has been deprecated by > > Elastic . > > - There is an `ApiCompatibilityMode` you can use in the latest ES 7 > > RestHighLevelClient to connect to an ES 8 cluster (works on my machine). > > - Between the 7.10 (used now) and the 7.17 (latest) they have introduced > in > > addition to the ApiCompatibilityMode also several breaking API changes by > > changing the packages of several classes we use. > > - If you update to the latest 7x then the > > `flink-connector-elasticsearch-base` can no longer be shared between ES 6 > > and ES 7 simply because of these package changes. > > - In 2022 there was still an ES 6 release but according to > > https://www.elastic.co/support/eol it is now EOL. > > > > So I now see several ways forward: > > 1) We drop ES6, update ES7 and use this compatibility mode as a quick > > solution. > > 2) We keep ES6, duplicating the `flink-connector-elasticsearch-base` > (into > > a 6.x and 7.x version with only these few classes being different), > update > > ES7 and use this compatibility mode as a quick solution. > > > > Later on someone should then create a connector based on the new > > ElasticSearch 8 Java client API. > > > > See also: > > > > > https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-compatibility.html > > > > > https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/master/migrate-hlrc.html > > > > My question is > > - Which do we do? > > - I vote for the "We drop ES6, update ES7 and use this compatibility > > mode" > > - Where do we do it? > > - We have these connectors in both the 'flink' and > > 'flink-connector-elasticsearch' repos. Do we update only the new one? Or > > both? > > > > > > -- > > Best regards / Met vriendelijke groeten, > > > > Niels Basjes > > > -- Best regards / Met vriendelijke groeten, Niels Basjes
