Re: Elasticsearch connector support?

Hi HangThanks for the link. I will wait for 3.1 connector release and hope it will be included. Med venlig hilsen / Best regardsLasse NedergaardDen 27. nov. 2023 kl. 12.00 skrev Hang Ruan :Hi, Lasse.There is already a discussion about the connector releases for 1.18[1].Best,Hang[1] https://lists.a

Re: Elasticsearch connector support?

Hi, Lasse. There is already a discussion about the connector releases for 1.18[1]. Best, Hang [1] https://lists.apache.org/thread/r31f988m57rtjy4s75030pzwrlqybpq2 Lasse Nedergaard 于2023年11月24日周五 22:57写道： > Hi > > From the documentation I can see there isn’t any ES support in Flink 1.18 > righ

Elasticsearch connector support?

Hi From the documentation I can see there isn’t any ES support in Flink 1.18 right now and Flink-26088 (ES 8 support) is still open. Does anyone has an idea when ES connector support will be available in 1.18 Please let me know. Med venlig hilsen / Best regards Lasse Nedergaard

Re: Elasticsearch Connector 1.17 not on maven central.

link >>> flink-connector-elasticsearch7 >>> 3.0.0-1.17 >>> >> >> >> It seems that currently the latest version for elasticsearch connector >> on maven central repository is 3.0.0-1.16( >> https://central.sonatype.com/artifact/org.apache.flink/flink-connector-elasticsearch7/3.0.0-1.16). >> I wonder if something was missed out. >> >> Thanks! >> >

Re: Elasticsearch Connector 1.17 not on maven central.

lease-1.17/docs/connectors/datastream/elasticsearch/, > maven can't find the required version: > >> >> org.apache.flink >> flink-connector-elasticsearch7 >> 3.0.0-1.17 >> > > > It seems that currently the latest version for elasticsear

Elasticsearch Connector 1.17 not on maven central.

earch7 > 3.0.0-1.17 > It seems that currently the latest version for elasticsearch connector on maven central repository is 3.0.0-1.16( https://central.sonatype.com/artifact/org.apache.flink/flink-connector-elasticsearch7/3.0.0-1.16). I wonder if something was missed out. Thanks!

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

nvolved! > > On Thu, Nov 10, 2022 at 12:51 PM Chesnay Schepler > wrote: > >> The Apache Flink community is very happy to announce the release of >> Apache Flink Elasticsearch Connector 3.0.0. >> >> Apache Flink® is an open-source stream processing framewo

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

with the first externalized connector for Flink. Thanks a lot to > all of you involved! > > On Thu, Nov 10, 2022 at 12:51 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: > The Apache Flink community is very happy to announce the release of > Apache Flink Elasticse

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

Really happy with the first externalized connector for Flink. Thanks a lot to all of you involved! On Thu, Nov 10, 2022 at 12:51 PM Chesnay Schepler wrote: > The Apache Flink community is very happy to announce the release of > Apache Flink Elasticsearch Connector 3.0.0. > > Apache

[ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Elasticsearch Connector 3.0.0. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available

Re: Flink Elasticsearch connector 7 with SSL

Hi Shishal, Please don't cross-post your question in multiple channels. As answered in Slack, SSL connections to Elasticsearch7 are currently not supported and are tracked under https://issues.apache.org/jira/browse/FLINK-27054 An Elasticsearch8 connector is being tracked under https://issues.apa

Flink Elasticsearch connector 7 with SSL

Hi Flink community members, In my flink job, I am trying to use *elasticsearch7* connector. I could not find a way to communicate over SSL when using *Elasticsearch7SinkBuilder* Am I missing something? Also there is an deprecated class *ElasticsearchSink.Builder* where it has *RestClientFactory*

Re: Number of parallel connections for Elasticsearch Connector

-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBase.java#L204 > . > > Best, > Yangze Guo > > On Sun, Jan 17, 2021 at 11:33 AM Rex Fenley wrote: > > > > I found the following, indicating that there is no concurrency for th

Re: Number of parallel connections for Elasticsearch Connector

On Sun, Jan 17, 2021 at 11:33 AM Rex Fenley wrote: > > I found the following, indicating that there is no concurrency for the > Elasticsearch Connector > https://github.com/apache/flink/blob/97bfd049951f8d52a2e0aed14265074c4255ead0/flink-connectors/flink-connector-elasticsearch-bas

Re: Number of parallel connections for Elasticsearch Connector

I found the following, indicating that there is no concurrency for the Elasticsearch Connector https://github.com/apache/flink/blob/97bfd049951f8d52a2e0aed14265074c4255ead0/flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch

Number of parallel connections for Elasticsearch Connector

Hello, How many connections does the ES connector use to write to Elasticsearch? We have a single machine with 16 vCPUs and parallelism of 4 running our job, with -p 4 I'd expect there to be 4 parallel bulk request writers / connections to Elasticsearch. Is there a place in the code to confirm thi

Re: ElasticSearch Connector

Wysakowicz > mailto:dwysakow...@apache.org>> wrote: > > Hi, > > I'm afraid you cannot write to different indices using the table > API ElasticSearch connector. Now I know why you wanted to go > throug

Re: Secure TLS/SSL ElasticSearch connector for current and future connector

Hi Christophe, Thanks so much for the pointers. That helps. Looking at the latest update on https://issues.apache.org/jira/browse/FLINK-8101 , there was an issue related to HLR retry handling. If I read this correctly, there is a bug in ES/HLR

Re: Secure TLS/SSL ElasticSearch connector for current and future connector

Hi Fritz, I think the High Level Rest Client implementation in this PR: https://github.com/apache/flink/pull/5374 should work. If you don't get the certificate properly available in your Java certs, you might want to redefine the createClient method to do something along those lines to get the con

Secure TLS/SSL ElasticSearch connector for current and future connector

Hi All, Anyone know if Flink has TLS/SSL support for the current ES connector ? If yes, any sample configuration/code ? If not, would TLS/SSL be support in the upcoming ES connector using Java High Level client ? Thanks, Fritz

Re: ElasticSearch Connector for version 6.x and scala 2.11

Actually, Flink's netty dependency (4.0.27) is shaded away into the "org.apache.flink.shaded.netty4.io.netty" package now (since version 1.4) and should thus not clash anymore. However, other netty versions may come into play from the job itself or from the integration of Hadoop's classpath (if ava

Re: ElasticSearch Connector for version 6.x and scala 2.11

Hi A workmate of mine tried to migrate the existing flink connector to ElasticSearch 6 but we had problems with netty dependencies that clashed (Flink uses 4.0.27 and ES is on 4.1). You can change the flink-connector-elasticsearch5 connector to ES 5.6.4, but then you have to do some adaptions to g

Re: ElasticSearch Connector for version 6.x and scala 2.11

Hi Rahul, Flink does not provide a connector for ElasticSearch 6 yet. There is this JIRA issue to track the development progress [1]. Best, Fabian [1] https://issues.apache.org/jira/browse/FLINK-8101 2017-12-01 7:22 GMT+01:00 Rahul Raj : > Hi All, > > Is there a Flink Elastic search connector

ElasticSearch Connector for version 6.x and scala 2.11

Hi All, Is there a Flink Elastic search connector for version 6.0 and scala 2.11? I couldn't find it listed here https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html . Regards, Rahul Raj

Re: Flink Elasticsearch Connector: Lucene Error message

all lucene and elasticsearch libs in your flink env and just use maven to manage dependencies, when working with the flink elasticsearch connector. so in the first place i deleted the libs in the folder to see if its working, but it did not. then we thought if maybe flink loads already the libs at

Re: Flink Elasticsearch Connector: Lucene Error message

TL;DR: remove all lucene and elasticsearch libs in your flink env and just use maven to manage dependencies, when working with the flink elasticsearch connector. so in the first place i deleted the libs in the folder to see if its working, but it did not. then we thought if maybe flink loads

Re: Flink Elasticsearch Connector: Lucene Error message

ters to each other, and i can telnet to the 9300 port of flink, so i think the connection is not an issue here.  We are currently using in our live env a custom elasticsearch connector, which used some extra lib's deployed on the cluster. i found one lucene lib and deleted it (since all depe

Re: Flink Elasticsearch Connector: Lucene Error message

>> Best, >> Aljoscha >> >> On 13. Jul 2017, at 13:56, Fabian Wollert >> wrote: >> >> Hi Timo, Hi Gordon, >> >> thx for the reply! I checked the connection from both clusters to each >> other, and i can telnet to the 9300 port of f

Re: Flink Elasticsearch Connector: Lucene Error message

o, Hi Gordon, >> >> thx for the reply! I checked the connection from both clusters to each >> other, and i can telnet to the 9300 port of flink, so i think the connection >> is not an issue here. >> >> We are currently using in our live env a custom elasticsearch conn

Re: Flink Elasticsearch Connector: Lucene Error message

he > connection is not an issue here. > > We are currently using in our live env a custom elasticsearch connector, > which used some extra lib's deployed on the cluster. i found one lucene lib > and deleted it (since all dependencies should be in the flink job jar), bu

Re: Flink Elasticsearch Connector: Lucene Error message

nk, so i think the connection is not > an issue here. > > We are currently using in our live env a custom elasticsearch connector, > which used some extra lib's deployed on the cluster. i found one lucene lib > and deleted it (since all dependencies should be in the flink job jar),

Re: Flink Elasticsearch Connector: Lucene Error message

Hi Timo, Hi Gordon, thx for the reply! I checked the connection from both clusters to each other, and i can telnet to the 9300 port of flink, so i think the connection is not an issue here. We are currently using in our live env a custom elasticsearch connector, which used some extra lib&#

Re: Flink Elasticsearch Connector: Lucene Error message

Hi Fabian, I loop in Gordon. Maybe he knows whats happening here. Regards, Timo Am 13.07.17 um 13:26 schrieb Fabian Wollert: Hi everyone, I'm trying to make use of the new Elasticsearch Connector <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsea

Flink Elasticsearch Connector: Lucene Error message

Hi everyone, I'm trying to make use of the new Elasticsearch Connector <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html>. I got a version running locally (with ssh tunnels to my Elasticsearch cluster in AWS) in my IDE, I see the data in El

Re: Elasticsearch connector and number of shards

I've just added a JIRA improvement ticket for this ( https://issues.apache.org/jira/browse/FLINK-4491). Best, Flavio On Wed, Jul 20, 2016 at 4:21 PM, Maximilian Michels wrote: > The connector doesn't cover this use case. Through the API you need to > use the IndicesAdminClient: > https://www.el

Re: Elasticsearch connector and number of shards

The connector doesn't cover this use case. Through the API you need to use the IndicesAdminClient: https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-admin-indices.html Otherwise Elasticsearch creates an index with shards automatically. We could add support for configuring

Re: Elasticsearch connector and number of shards

Indeed, we've tried with the parameter *index.number_of_shards* but it didn't work so I fear that this parameter is not handled by the current implementation..am I wrong? On Mon, Jul 18, 2016 at 11:37 AM, Ufuk Celebi wrote: > I've never used the Elasticsearch sink, but the docs say: > > "Note ho

Re: Elasticsearch connector and number of shards

I've never used the Elasticsearch sink, but the docs say: "Note how a Map of Strings is used to configure the Sink. The configuration keys are documented in the Elasticsearch documentationhere. Especially important is the cluster.name parameter that must correspond to the name of your cluster." T

Elasticsearch connector and number of shards

Hi to all, we tried to use the streaming ES connector of Flink 1.1-SNAPSHOT and we wanted to set the number of shards when creating a new index. Is that possible at the moment? Best, Flavio

Re: Elasticsearch Connector

, Till ​ On Thu, Jun 16, 2016 at 5:14 PM, Eamon Kavanagh wrote: > Hey Support, > > I'm trying to use Flink's Elasticsearch connector but I'm having trouble. > When I add the dependency seen here ( > https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/

Elasticsearch Connector

Hey Support, I'm trying to use Flink's Elasticsearch connector but I'm having trouble. When I add the dependency seen here ( https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/connectors/elasticsearch2.html) to my pom file, IntelliJ can't find it. I also

Re: Any way for Flink Elasticsearch connector reflecting IP change of Elasticsearch cluster?

same feature (reconnect) be useful for Kafka >>> connector ? >>> For example, if the IP of broker changes. >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-flink-user-mailing-list-archive.2336050.n4.na

Re: Any way for Flink Elasticsearch connector reflecting IP change of Elasticsearch cluster?

View this message in context: >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Any-way-for-Flink-Elasticsearch-connector-reflecting-IP-change-of-Elasticsearch-cluster-tp6597p6688.html >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive at Nabble.com. >> > >

Re: Any way for Flink Elasticsearch connector reflecting IP change of Elasticsearch cluster?

? > For example, if the IP of broker changes. > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Any-way-for-Flink-Elasticsearch-connector-reflecting-IP-change-of-Elasticsearch-cluster-tp6597p6688.html > Sent from

Re: Any way for Flink Elasticsearch connector reflecting IP change of Elasticsearch cluster?

Glad to see it's developing. Can I ask would the same feature (reconnect) be useful for Kafka connector ? For example, if the IP of broker changes. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Any-way-for-Flink-Elasticsearch-conn

Re: Any way for Flink Elasticsearch connector reflecting IP change of Elasticsearch cluster?

? 2016-05-02 9:38 GMT+02:00 Sendoh : > Hi, > > When using Elasticsearch connector, Is there a way to reflect IP change of > Elasticsearch cluster? > We use DNS of Elasticsearch in data sink, e.g. elasticsearch-dev.foo.de. > However, when we replace the old Elasticsearch cluster w

Any way for Flink Elasticsearch connector reflecting IP change of Elasticsearch cluster?

Hi, When using Elasticsearch connector, Is there a way to reflect IP change of Elasticsearch cluster? We use DNS of Elasticsearch in data sink, e.g. elasticsearch-dev.foo.de. However, when we replace the old Elasticsearch cluster with a new one, the Elasticsearch connector cannot write into the

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

> shouldn't be better to have both connectors for ES?one for 1.x and another > for 2.x? IMHO that's the way to go. Thanks Madhukar! Cheers, Max On Sat, Dec 5, 2015 at 6:49 AM, Deepak Sharma wrote: > Hi Madhu > Would you be able to provide the use case here in ElasticSearch with Flink? > > Th

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

Hi Madhu Would you be able to provide the use case here in ElasticSearch with Flink? Thanks Deepak On Sat, Dec 5, 2015 at 1:25 AM, Madhukar Thota wrote: > Sure. I can submit the pull request. > > On Fri, Dec 4, 2015 at 12:37 PM, Maximilian Michels > wrote: > >> Hi Madhu, >> >> Great. Do you wa

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

shouldn't be better to have both connectors for ES?one for 1.x and another for 2.x? On 4 Dec 2015 20:55, "Madhukar Thota" wrote: > Sure. I can submit the pull request. > > On Fri, Dec 4, 2015 at 12:37 PM, Maximilian Michels > wrote: > >> Hi Madhu, >> >> Great. Do you want to contribute it back v

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

Sure. I can submit the pull request. On Fri, Dec 4, 2015 at 12:37 PM, Maximilian Michels wrote: > Hi Madhu, > > Great. Do you want to contribute it back via a GitHub pull request? If > not that's also fine. We will try look into the 2.0 connector next > week. > > Best, > Max > > On Fri, Dec 4, 2

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

Hi Madhu, Great. Do you want to contribute it back via a GitHub pull request? If not that's also fine. We will try look into the 2.0 connector next week. Best, Max On Fri, Dec 4, 2015 at 4:16 PM, Madhukar Thota wrote: > i have created working connector for Elasticsearch 2.0 based on > elasticse

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

i have created working connector for Elasticsearch 2.0 based on elasticsearch-flink connector. I am using it right now but i want official connector from flink. ElasticsearchSink.java import org.apache.flink.api.java.utils.ParameterTool; import org.apache.flink.configuration.Configuration; impor

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

Hi Madhu, Not yet. The API has changed slightly. We'll add one very soon. In the meantime I've created an issue to keep track of the status: https://issues.apache.org/jira/browse/FLINK-3115 Thanks, Max On Thu, Dec 3, 2015 at 10:50 PM, Madhukar Thota wrote: > is current elasticsearch-flink conn

Flink-Elasticsearch connector support for elasticsearch 2.0

is current elasticsearch-flink connector support elasticsearch 2.x version? -Madhu