+1 for removing ES2 and updating ES5 in 1.11. +1 for revisiting the removal of ES5 for 1.12.
Cheers, Till On Tue, Feb 18, 2020 at 1:28 PM Chesnay Schepler <ches...@apache.org> wrote: > Since one of the reasons for dropping ES2 was that it blocks some > critical updates for the ES5 connector I'd prefer to keep ES5 around for > 1.11, and revisit this discussion for 1.12 . > > On 18/02/2020 13:03, Aljoscha Krettek wrote: > > Wouldn't removing the ES 2.x connector be enough because we can then > > update the ES 5.x connector? It seems there are some users that still > > want to use that one. > > > > Best, > > Aljoscha > > > > On 18.02.20 10:42, Robert Metzger wrote: > >> The ES5 connector is causing some problems on the CI system. It would be > >> nice if we could make a decision here soon. I don't want to invest time > >> into fixing it, if we are going to remove it. > >> > >> I'm still in favor of removing it. If we see that there's demand for the > >> 5.x connector after the 1.11 release, somebody can take the source and > >> contribute it to Apache Bahir or a GitHub account and then posts it to > >> flink-packages.org. > >> > >> > >> On Thu, Feb 13, 2020 at 3:34 PM Dawid Wysakowicz > >> <dwysakow...@apache.org> > >> wrote: > >> > >>> Sorry for late reply, > >>> > >>> @all I think there is a general consensus that we want to drop ES 2.x > >>> support. I created https://issues.apache.org/jira/browse/FLINK-16046 > to > >>> track it. > >>> > >>> > >>> @Stephan @Chesnay @Itamar In our connectors we use Java High Level Rest > >>> Client. ES promises to maintain compatibility of it with any newer > >>> minor > >>> version of ES. So if we have 6.1 client we can use it with any 6.2, 6.3 > >>> etc. > >>> > >>> ES provides also a low level rest client which does not include any > >>> direct es dependencies and can work with any version of ES. It does not > >>> provide any marshalling unmarshalling or higher level features as > >>> Chesnay said. > >>> > >>> Correct me if I am wrong @Itamar but your HTTP client is a simplified > >>> version of the ES's high level rest client with a subset of its > >>> features. I think it will still have the same problems as ES's High > >>> Level Rest Client's because ES does not guarantee that newer message > >>> formats will be compatible with older versions of ES or that message > >>> formats are compatible across major versions at all. > >>> > >>> > >>> @Stephan @Danny As for the 5.x connector. Any ideas how can we get > >>> user's feedback about it? I cross posted on the user mailing list with > >>> no luck so far. Personally I would be in favor of dropping the > >>> connector. Worst case scenario users still have the possibility of > >>> building the connector themselves from source with just bumping the > >>> flink's versions. As far as I can tell there were no changes to the > >>> code > >>> base for quite some time. > >>> > >>> Best, > >>> > >>> Dawid > >>> > >>> On 11/02/2020 10:46, Chesnay Schepler wrote: > >>>> I suppose the downside in an HTTP ES sink is that you don't get _any_ > >>>> form of high-level API from ES, and we'd have to manually build an > >>>> HTTP request that matches the ES format. Of course you also lose any > >>>> client-side verification that the clients did, if there is any (but I > >>>> guess the API itself prevented certain errors). > >>>> > >>>> On 11/02/2020 09:32, Stephan Ewen wrote: > >>>>> +1 to drop ES 2.x - unsure about 5.x (makes sense to get more user > >>>>> input > >>>>> for that one). > >>>>> > >>>>> @Itamar - if you would be interested in contributing a "universal" or > >>>>> "cross version" ES connector, that could be very interesting. Do you > >>>>> know > >>>>> if there are known performance issues or feature restrictions with > >>>>> that > >>>>> approach? > >>>>> @dawid what do you think about that? > >>>>> > >>>>> > >>>>> On Tue, Feb 11, 2020 at 6:28 AM Danny Chan <yuzhao....@gmail.com> > >>> wrote: > >>>>> > >>>>>> 5.x seems to have a lot of users, is the 6.x completely > >>>>>> compatible with > >>>>>> 5.x ~ > >>>>>> > >>>>>> Best, > >>>>>> Danny Chan > >>>>>> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz > >>>>>> <dwysakow...@apache.org>，写道： > >>>>>>> Hi all, > >>>>>>> > >>>>>>> As described in this > >>> https://issues.apache.org/jira/browse/FLINK-11720 > >>>>>>> ticket our elasticsearch 5.x connector does not work out of the > >>>>>>> box on > >>>>>>> some systems and requires a version bump. This also happens for our > >>>>>>> e2e. > >>>>>>> We cannot bump the version in es 5.x connector, because 5.x > >>>>>>> connector > >>>>>>> shares a common class with 2.x that uses an API that was replaced > >>>>>>> in 5.2. > >>>>>>> > >>>>>>> Both versions are already long eol: > >>> https://www.elastic.co/support/eol > >>>>>>> > >>>>>>> I suggest to drop both connectors 5.x and 2.x. If it is too much to > >>>>>>> drop > >>>>>>> both of them, I would strongly suggest dropping at least 2.x > >>>>>>> connector > >>>>>>> and update the 5.x line to a working es client module. > >>>>>>> > >>>>>>> What do you think? Should we drop both versions? Drop only the 2.x > >>>>>>> connector? Or keep them both? > >>>>>>> > >>>>>>> Best, > >>>>>>> > >>>>>>> Dawid > >>>>>>> > >>>>>>> > >>>> > >>> > >>> > >> > > > >
