Hi, As part of restructuring the ES connectors, I think it also a very good timing to unify how our ES connectors are used.
For the 1.x connector, users provide a “IndexRequestBuilder” that was only capable of indexing a single document for each record. For the 2.x and pending 5.x connector, the API was improved to let users provide a “ElasticsearchSinkFunction” that can yield many ES actions (modify, index, delete, etc.) for each record. I’m not sure of the history on this, but it seems like the API improvement to 2.x+ wasn’t reflected also on the 1.x connector at the time. I’d like to deprecate the “IndexRequestBuilder” constructor variant in 1.x, and let it have a new constructor that takes “ElasticsearchSinkFunction” like the other versions. This will also have the advantage of much cleaner structure across the ES connector modules. What do you think? - Gordon On January 9, 2017 at 10:50:55 PM, Kostas Tzoumas (ktzou...@apache.org) wrote: +1 for keeping the ES connector in Flink On Mon, Jan 9, 2017 at 2:53 PM, Robert Metzger <rmetz...@apache.org> wrote: > Thanks for bringing this up again Flavio. > > The good thing about a common base for all ES connectors is that we can fix > the issue for all ES versions (ideally). > > Thanks a lot for taking care of my proposed changes Gordon! > > > On Mon, Jan 9, 2017 at 12:47 PM, Flavio Pompermaier <pomperma...@okkam.it> > wrote: > > > I think also https://issues.apache.org/jira/browse/FLINK-5353 should be > > solved sooner or later... > > > > Best, > > Flavio > > > > On 9 Jan 2017 12:24, "Tzu-Li (Gordon) Tai" <tzuli...@apache.org> wrote: > > > > > Thanks for starting the discussion Robert. > > > > > > I agree that we should try to treat the Elasticsearch connectors in a > > > similar fashion to the multiple Kafka connector versions. > > > > > > +1 for flink-connector-elasticsearch-base > > > +1 to consolidating the documents > > > +1 for keeping the ES connector in Flink > > > > > > I’d also like to proceed with this with https://github.com/apache/ > > > flink/pull/2767, by opening another new PR based on #2767 that also > > > refactors the ES connectors & consolidates documentation. > > > After that, we can proceed with https://github.com/apache/flin > > k/pull/2861 for > > > the proper at least once support across all ES versions. > > > > > > - Gordon > > > > > > On December 19, 2016 at 11:58:03 PM, Fabian Hueske (fhue...@gmail.com) > > > wrote: > > > > > > Thanks for looking into this Robert. > > > I agree, it would be good to share as much code as possible between the > > > Elasticsearch connectors. > > > > > > +1 for flink-connector-elasticsearch-base > > > +1 for keeping the connectors in Flink > > > > > > 2016-12-19 16:46 GMT+01:00 Robert Metzger <rmetz...@apache.org>: > > > > > > > Hi, > > > > I'm wondering how we want to maintain the ElasticSearch sinks in > Flink. > > > > > > > > Right now, we have two connectors, supporting the 1.x and 2.x > versions. > > > > There is a pending PR [1] for adding ES 5 support. > > > > > > > > All three connectors share a lot of code, and there is a pull request > > [2] > > > > improving the ES 2.x connector. If we want to make the ES sinks have > > > proper > > > > at least once support, we need to do some more changes, and I would > > like > > > to > > > > do them once for all connectors. > > > > I propose to create a new "flink-connector-elasticsearch-base" > module > > > with > > > > the common code and then create modules for 1.x, 2.x and 5.x. > > > > Also, I would merge the documentation pages into one. > > > > > > > > I would also be open to discuss whether we want to move the EL > > connectors > > > > into Bahir. > > > > For now, I'm against that, given that ES is among the more popular > > > > connectors in Flink. > > > > > > > > > > > > Regards, > > > > Robert > > > > > > > > [1] https://github.com/apache/flink/pull/2767 > > > > [2] https://github.com/apache/flink/pull/2861 > > > > > > > > > >
