Thank you all. I have proposed a PR to upgrade the ES5 connector: https://github.com/apache/flink/pull/11162
On Wed, Feb 19, 2020 at 3:02 PM Till Rohrmann <trohrm...@apache.org> wrote: > +1 for removing ES2 and updating ES5 in 1.11. > > +1 for revisiting the removal of ES5 for 1.12. > > Cheers, > Till > > On Tue, Feb 18, 2020 at 1:28 PM Chesnay Schepler <ches...@apache.org> > wrote: > > > Since one of the reasons for dropping ES2 was that it blocks some > > critical updates for the ES5 connector I'd prefer to keep ES5 around for > > 1.11, and revisit this discussion for 1.12 . > > > > On 18/02/2020 13:03, Aljoscha Krettek wrote: > > > Wouldn't removing the ES 2.x connector be enough because we can then > > > update the ES 5.x connector? It seems there are some users that still > > > want to use that one. > > > > > > Best, > > > Aljoscha > > > > > > On 18.02.20 10:42, Robert Metzger wrote: > > >> The ES5 connector is causing some problems on the CI system. It would > be > > >> nice if we could make a decision here soon. I don't want to invest > time > > >> into fixing it, if we are going to remove it. > > >> > > >> I'm still in favor of removing it. If we see that there's demand for > the > > >> 5.x connector after the 1.11 release, somebody can take the source and > > >> contribute it to Apache Bahir or a GitHub account and then posts it to > > >> flink-packages.org. > > >> > > >> > > >> On Thu, Feb 13, 2020 at 3:34 PM Dawid Wysakowicz > > >> <dwysakow...@apache.org> > > >> wrote: > > >> > > >>> Sorry for late reply, > > >>> > > >>> @all I think there is a general consensus that we want to drop ES 2.x > > >>> support. I created https://issues.apache.org/jira/browse/FLINK-16046 > > to > > >>> track it. > > >>> > > >>> > > >>> @Stephan @Chesnay @Itamar In our connectors we use Java High Level > Rest > > >>> Client. ES promises to maintain compatibility of it with any newer > > >>> minor > > >>> version of ES. So if we have 6.1 client we can use it with any 6.2, > 6.3 > > >>> etc. > > >>> > > >>> ES provides also a low level rest client which does not include any > > >>> direct es dependencies and can work with any version of ES. It does > not > > >>> provide any marshalling unmarshalling or higher level features as > > >>> Chesnay said. > > >>> > > >>> Correct me if I am wrong @Itamar but your HTTP client is a simplified > > >>> version of the ES's high level rest client with a subset of its > > >>> features. I think it will still have the same problems as ES's High > > >>> Level Rest Client's because ES does not guarantee that newer message > > >>> formats will be compatible with older versions of ES or that message > > >>> formats are compatible across major versions at all. > > >>> > > >>> > > >>> @Stephan @Danny As for the 5.x connector. Any ideas how can we get > > >>> user's feedback about it? I cross posted on the user mailing list > with > > >>> no luck so far. Personally I would be in favor of dropping the > > >>> connector. Worst case scenario users still have the possibility of > > >>> building the connector themselves from source with just bumping the > > >>> flink's versions. As far as I can tell there were no changes to the > > >>> code > > >>> base for quite some time. > > >>> > > >>> Best, > > >>> > > >>> Dawid > > >>> > > >>> On 11/02/2020 10:46, Chesnay Schepler wrote: > > >>>> I suppose the downside in an HTTP ES sink is that you don't get > _any_ > > >>>> form of high-level API from ES, and we'd have to manually build an > > >>>> HTTP request that matches the ES format. Of course you also lose any > > >>>> client-side verification that the clients did, if there is any (but > I > > >>>> guess the API itself prevented certain errors). > > >>>> > > >>>> On 11/02/2020 09:32, Stephan Ewen wrote: > > >>>>> +1 to drop ES 2.x - unsure about 5.x (makes sense to get more user > > >>>>> input > > >>>>> for that one). > > >>>>> > > >>>>> @Itamar - if you would be interested in contributing a "universal" > or > > >>>>> "cross version" ES connector, that could be very interesting. Do > you > > >>>>> know > > >>>>> if there are known performance issues or feature restrictions with > > >>>>> that > > >>>>> approach? > > >>>>> @dawid what do you think about that? > > >>>>> > > >>>>> > > >>>>> On Tue, Feb 11, 2020 at 6:28 AM Danny Chan <yuzhao....@gmail.com> > > >>> wrote: > > >>>>> > > >>>>>> 5.x seems to have a lot of users, is the 6.x completely > > >>>>>> compatible with > > >>>>>> 5.x ~ > > >>>>>> > > >>>>>> Best, > > >>>>>> Danny Chan > > >>>>>> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz > > >>>>>> <dwysakow...@apache.org>，写道： > > >>>>>>> Hi all, > > >>>>>>> > > >>>>>>> As described in this > > >>> https://issues.apache.org/jira/browse/FLINK-11720 > > >>>>>>> ticket our elasticsearch 5.x connector does not work out of the > > >>>>>>> box on > > >>>>>>> some systems and requires a version bump. This also happens for > our > > >>>>>>> e2e. > > >>>>>>> We cannot bump the version in es 5.x connector, because 5.x > > >>>>>>> connector > > >>>>>>> shares a common class with 2.x that uses an API that was replaced > > >>>>>>> in 5.2. > > >>>>>>> > > >>>>>>> Both versions are already long eol: > > >>> https://www.elastic.co/support/eol > > >>>>>>> > > >>>>>>> I suggest to drop both connectors 5.x and 2.x. If it is too much > to > > >>>>>>> drop > > >>>>>>> both of them, I would strongly suggest dropping at least 2.x > > >>>>>>> connector > > >>>>>>> and update the 5.x line to a working es client module. > > >>>>>>> > > >>>>>>> What do you think? Should we drop both versions? Drop only the > 2.x > > >>>>>>> connector? Or keep them both? > > >>>>>>> > > >>>>>>> Best, > > >>>>>>> > > >>>>>>> Dawid > > >>>>>>> > > >>>>>>> > > >>>> > > >>> > > >>> > > >> > > > > > > > >
