Andriy Redko created FLINK-25756:
------------------------------------
Summary: Dedicated Opensearch connectors
Key: FLINK-25756
URL: https://issues.apache.org/jira/browse/FLINK-25756
Project: Flink
Issue Type: Improvement
Reporter: Andriy Redko
Since the time Opensearch got forked from Elasticsearch a few things got
changed. The projects evolve in different directions, the Elasticsearch clients
up to 7.13.x were able to connect to Opensearch clusters, but since 7.14 - not
anymore [1] (Elastic continues to harden their clients to connect to
Elasticsearch clusters only).
For example, running current Flink master against Opensearch clusters using
Elalsticsearch 7 connectors would fail with:
```
{color:#008080}Caused{color}{color:#000000}
{color}{color:#0000ff}by{color}{color:#000000}:
{color}{color:#008080}ElasticsearchException{color}{color:#000000}[{color}{color:#008080}Elasticsearch{color}{color:#000000}
version {color}{color:#098658}6{color}{color:#000000}
{color}{color:#0000ff}or{color}{color:#000000} more is required]{color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.{color}{color:#008080}RestHighLevelClient$4{color}{color:#000000}.onResponse({color}{color:#008080}RestHighLevelClient{color}{color:#000000}.java:{color}{color:#098658}2056{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.{color}{color:#008080}RestHighLevelClient$4{color}{color:#000000}.onResponse({color}{color:#008080}RestHighLevelClient{color}{color:#000000}.java:{color}{color:#098658}2043{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.common.util.concurrent.{color}{color:#008080}ListenableFuture{color}{color:#000000}.notifyListenerDirectly({color}{color:#008080}ListenableFuture{color}{color:#000000}.java:{color}{color:#098658}113{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.common.util.concurrent.{color}{color:#008080}ListenableFuture{color}{color:#000000}.done({color}{color:#008080}ListenableFuture{color}{color:#000000}.java:{color}{color:#098658}100{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.common.util.concurrent.{color}{color:#008080}BaseFuture{color}{color:#000000}.{color}{color:#0000ff}set{color}{color:#000000}({color}{color:#008080}BaseFuture{color}{color:#000000}.java:{color}{color:#098658}133{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.common.util.concurrent.{color}{color:#008080}ListenableFuture{color}{color:#000000}.onResponse({color}{color:#008080}ListenableFuture{color}{color:#000000}.java:{color}{color:#098658}139{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.{color}{color:#008080}RestHighLevelClient$5{color}{color:#000000}.onSuccess({color}{color:#008080}RestHighLevelClient{color}{color:#000000}.java:{color}{color:#098658}2129{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.{color}{color:#008080}RestClient$FailureTrackingResponseListener{color}{color:#000000}.onSuccess({color}{color:#008080}RestClient{color}{color:#000000}.java:{color}{color:#098658}636{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.{color}{color:#008080}RestClient$1{color}{color:#000000}.completed({color}{color:#008080}RestClient{color}{color:#000000}.java:{color}{color:#098658}376{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.{color}{color:#008080}RestClient$1{color}{color:#000000}.completed({color}{color:#008080}RestClient{color}{color:#000000}.java:{color}{color:#098658}370{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.concurrent.{color}{color:#008080}BasicFuture{color}{color:#000000}.completed({color}{color:#008080}BasicFuture{color}{color:#000000}.java:{color}{color:#098658}122{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.client.{color}{color:#008080}DefaultClientExchangeHandlerImpl{color}{color:#000000}.responseCompleted({color}{color:#008080}DefaultClientExchangeHandlerImpl{color}{color:#000000}.java:{color}{color:#098658}181{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.nio.protocol.{color}{color:#008080}HttpAsyncRequestExecutor{color}{color:#000000}.processResponse({color}{color:#008080}HttpAsyncRequestExecutor{color}{color:#000000}.java:{color}{color:#098658}448{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.nio.protocol.{color}{color:#008080}HttpAsyncRequestExecutor{color}{color:#000000}.inputReady({color}{color:#008080}HttpAsyncRequestExecutor{color}{color:#000000}.java:{color}{color:#098658}338{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.{color}{color:#008080}DefaultNHttpClientConnection{color}{color:#000000}.consumeInput({color}{color:#008080}DefaultNHttpClientConnection{color}{color:#000000}.java:{color}{color:#098658}265{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.client.{color}{color:#008080}InternalIODispatch{color}{color:#000000}.onInputReady({color}{color:#008080}InternalIODispatch{color}{color:#000000}.java:{color}{color:#098658}81{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.client.{color}{color:#008080}InternalIODispatch{color}{color:#000000}.onInputReady({color}{color:#008080}InternalIODispatch{color}{color:#000000}.java:{color}{color:#098658}39{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.{color}{color:#008080}AbstractIODispatch{color}{color:#000000}.inputReady({color}{color:#008080}AbstractIODispatch{color}{color:#000000}.java:{color}{color:#098658}114{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.{color}{color:#008080}BaseIOReactor{color}{color:#000000}.readable({color}{color:#008080}BaseIOReactor{color}{color:#000000}.java:{color}{color:#098658}162{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.{color}{color:#008080}AbstractIOReactor{color}{color:#000000}.processEvent({color}{color:#008080}AbstractIOReactor{color}{color:#000000}.java:{color}{color:#098658}337{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.{color}{color:#008080}AbstractIOReactor{color}{color:#000000}.processEvents({color}{color:#008080}AbstractIOReactor{color}{color:#000000}.java:{color}{color:#098658}315{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.{color}{color:#008080}AbstractIOReactor{color}{color:#000000}.execute({color}{color:#008080}AbstractIOReactor{color}{color:#000000}.java:{color}{color:#098658}276{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.{color}{color:#008080}BaseIOReactor{color}{color:#000000}.execute({color}{color:#008080}BaseIOReactor{color}{color:#000000}.java:{color}{color:#098658}104{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.reactor.{color}{color:#008080}AbstractMultiworkerIOReactor$Worker{color}{color:#000000}.run({color}{color:#008080}AbstractMultiworkerIOReactor{color}{color:#000000}.java:{color}{color:#098658}591{color}{color:#000000}){color}
```
With the compatibility mode [2] turned on, still fails further the line:
```
{color:#008080}Caused{color}{color:#000000}
{color}{color:#0000ff}by{color}{color:#000000}:
{color}{color:#008080}ElasticsearchException{color}{color:#000000}[{color}{color:#008080}Invalid{color}{color:#000000}
{color}{color:#0000ff}or{color}{color:#000000} missing tagline
[{color}{color:#008080}The{color}{color:#000000}
{color}{color:#008080}OpenSearch{color}{color:#000000}
{color}{color:#008080}Project{color}{color:#000000}:
https://opensearch.org/]]{color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.{color}{color:#008080}RestHighLevelClient$4{color}{color:#000000}.onResponse({color}{color:#008080}RestHighLevelClient{color}{color:#000000}.java:{color}{color:#098658}2056{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.{color}{color:#008080}RestHighLevelClient$4{color}{color:#000000}.onResponse({color}{color:#008080}RestHighLevelClient{color}{color:#000000}.java:{color}{color:#098658}2043{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.common.util.concurrent.{color}{color:#008080}ListenableFuture{color}{color:#000000}.notifyListenerDirectly({color}{color:#008080}ListenableFuture{color}{color:#000000}.java:{color}{color:#098658}113{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.common.util.concurrent.{color}{color:#008080}ListenableFuture{color}{color:#000000}.done({color}{color:#008080}ListenableFuture{color}{color:#000000}.java:{color}{color:#098658}100{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.common.util.concurrent.{color}{color:#008080}BaseFuture{color}{color:#000000}.{color}{color:#0000ff}set{color}{color:#000000}({color}{color:#008080}BaseFuture{color}{color:#000000}.java:{color}{color:#098658}133{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.common.util.concurrent.{color}{color:#008080}ListenableFuture{color}{color:#000000}.onResponse({color}{color:#008080}ListenableFuture{color}{color:#000000}.java:{color}{color:#098658}139{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.{color}{color:#008080}RestHighLevelClient$5{color}{color:#000000}.onSuccess({color}{color:#008080}RestHighLevelClient{color}{color:#000000}.java:{color}{color:#098658}2129{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.{color}{color:#008080}RestClient$FailureTrackingResponseListener{color}{color:#000000}.onSuccess({color}{color:#008080}RestClient{color}{color:#000000}.java:{color}{color:#098658}636{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.{color}{color:#008080}RestClient$1{color}{color:#000000}.completed({color}{color:#008080}RestClient{color}{color:#000000}.java:{color}{color:#098658}376{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.elasticsearch.client.{color}{color:#008080}RestClient$1{color}{color:#000000}.completed({color}{color:#008080}RestClient{color}{color:#000000}.java:{color}{color:#098658}370{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.concurrent.{color}{color:#008080}BasicFuture{color}{color:#000000}.completed({color}{color:#008080}BasicFuture{color}{color:#000000}.java:{color}{color:#098658}122{color}{color:#000000}){color}
{color:#000000} at
org.apache.flink.elasticsearch7.shaded.org.apache.http.impl.nio.client.{color}{color:#008080}DefaultClientExchangeHandlerImpl{color}{color:#000000}.responseCompleted{color}
```
Arguably, the best way to proceed in this situation is to provide dedicated
Opensearch connectors, on par with Elasticsearch ones.
Me ([~reta]) and [~Sergey Nuyanzin] would be more than happy to make the
contribution (and maintain it as well), if there are no objections from
committers / PMC members.
[1] [https://github.com/elastic/elasticsearch/pull/73910]
[2]
https://opensearch.org/docs/latest/upgrade-to/upgrade-to/#upgrade-elasticsearch-oss
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
