I definitively agree, if we can cover the functionality with all ES dependency versions.
On Tue, Jan 10, 2017 at 6:50 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > Hi, > > As part of restructuring the ES connectors, I think it also a very good > timing to unify how our ES connectors are used. > > For the 1.x connector, users provide a “IndexRequestBuilder” that was only > capable of indexing a single document for each record. > For the 2.x and pending 5.x connector, the API was improved to let users > provide a “ElasticsearchSinkFunction” that can yield many ES actions > (modify, index, delete, etc.) for each record. > > I’m not sure of the history on this, but it seems like the API improvement > to 2.x+ wasn’t reflected also on the 1.x connector at the time. > > I’d like to deprecate the “IndexRequestBuilder” constructor variant in > 1.x, and let it have a new constructor that takes > “ElasticsearchSinkFunction” like the other versions. > This will also have the advantage of much cleaner structure across the ES > connector modules. What do you think? > > - Gordon > > On January 9, 2017 at 10:50:55 PM, Kostas Tzoumas (ktzou...@apache.org) > wrote: > > +1 for keeping the ES connector in Flink > > On Mon, Jan 9, 2017 at 2:53 PM, Robert Metzger <rmetz...@apache.org> > wrote: > > > Thanks for bringing this up again Flavio. > > > > The good thing about a common base for all ES connectors is that we can > fix > > the issue for all ES versions (ideally). > > > > Thanks a lot for taking care of my proposed changes Gordon! > > > > > > On Mon, Jan 9, 2017 at 12:47 PM, Flavio Pompermaier < > pomperma...@okkam.it> > > wrote: > > > > > I think also https://issues.apache.org/jira/browse/FLINK-5353 should > be > > > solved sooner or later... > > > > > > Best, > > > Flavio > > > > > > On 9 Jan 2017 12:24, "Tzu-Li (Gordon) Tai" <tzuli...@apache.org> > wrote: > > > > > > > Thanks for starting the discussion Robert. > > > > > > > > I agree that we should try to treat the Elasticsearch connectors in a > > > > similar fashion to the multiple Kafka connector versions. > > > > > > > > +1 for flink-connector-elasticsearch-base > > > > +1 to consolidating the documents > > > > +1 for keeping the ES connector in Flink > > > > > > > > I’d also like to proceed with this with https://github.com/apache/ > > > > flink/pull/2767, by opening another new PR based on #2767 that also > > > > refactors the ES connectors & consolidates documentation. > > > > After that, we can proceed with https://github.com/apache/flin > > > k/pull/2861 for > > > > the proper at least once support across all ES versions. > > > > > > > > - Gordon > > > > > > > > On December 19, 2016 at 11:58:03 PM, Fabian Hueske ( > fhue...@gmail.com) > > > > wrote: > > > > > > > > Thanks for looking into this Robert. > > > > I agree, it would be good to share as much code as possible between > the > > > > Elasticsearch connectors. > > > > > > > > +1 for flink-connector-elasticsearch-base > > > > +1 for keeping the connectors in Flink > > > > > > > > 2016-12-19 16:46 GMT+01:00 Robert Metzger <rmetz...@apache.org>: > > > > > > > > > Hi, > > > > > I'm wondering how we want to maintain the ElasticSearch sinks in > > Flink. > > > > > > > > > > Right now, we have two connectors, supporting the 1.x and 2.x > > versions. > > > > > There is a pending PR [1] for adding ES 5 support. > > > > > > > > > > All three connectors share a lot of code, and there is a pull > request > > > [2] > > > > > improving the ES 2.x connector. If we want to make the ES sinks > have > > > > proper > > > > > at least once support, we need to do some more changes, and I would > > > like > > > > to > > > > > do them once for all connectors. > > > > > I propose to create a new "flink-connector-elasticsearch-base" > > module > > > > with > > > > > the common code and then create modules for 1.x, 2.x and 5.x. > > > > > Also, I would merge the documentation pages into one. > > > > > > > > > > I would also be open to discuss whether we want to move the EL > > > connectors > > > > > into Bahir. > > > > > For now, I'm against that, given that ES is among the more popular > > > > > connectors in Flink. > > > > > > > > > > > > > > > Regards, > > > > > Robert > > > > > > > > > > [1] https://github.com/apache/flink/pull/2767 > > > > > [2] https://github.com/apache/flink/pull/2861 > > > > > > > > > > > > > > >
