Hi Christophe, Thanks for bringing this up.
Yes, the main issue with the existing PRs and preventing it from moving forward is how it currently breaks initial assumptions of APIs in the `elasticsearch-base` module. Working around that would require introducing a new base module specifically for 5.3+ and 6.x+, which we've also agreed on the PR isn't a nice way to go. I had a quick stab at the REST API, and it seems to be promising, especially given that you mentioned that starting from next versions, the current API we use will be fully removed. I'm definitely a +1 to try to move this forward with a proper fix. Some other remarks / questions I have: - Maybe we can consider removing support for ES 1.x and 2.x starting from 1.6. Those are very old ES versions (considering that ES 6.x has already been out for a while). Do you think this would simply how our base module APIs are designed? - Wouldn't it be possible to have a REST implementation of the `ElasticsearchSinkCallBridge` for 5.x (covering both 5.2 and 5.3+)? If so, once we remove ES 1.x and 2.x, it might actually be possible to completely replace the current `elasticsearch-base` module. Cheers, Gordon On Sun, May 13, 2018 at 12:36 AM, Flavio Pompermaier <pomperma...@okkam.it> wrote: > +1. Torally agree > > On Sat, 12 May 2018, 18:14 Christophe Jolif, <cjo...@gmail.com> wrote: > > > Hi all, > > > > There is quite some time Flink Elasticsearch sink is broken for > > Elastisearch 5.x (nearly a year): > > > > https://issues.apache.org/jira/browse/FLINK-7386 > > > > And there is no support for Elasticsearch 6.x: > > > > https://issues.apache.org/jira/browse/FLINK-8101 > > > > However several PRs were issued: > > > > https://github.com/apache/flink/pull/4675 > > https://github.com/apache/flink/pull/5374 > > > > I also raised the issue on the mailing list in the 1.5 timeframe: > > > > http://apache-flink-mailing-list-archive.1008284.n3. > > nabble.com/DISCUSS-Releasing-Flink-1-5-0-td20867.html#a20905 > > > > But things are still not really moving. However this seems something > people > > are looking for, so I would really like the community to consider that > for > > 1.6. > > > > The problems I see from comments on the PRs: > > > > - getting something that is following the Flink APIs initially created > is a > > nightmare because Elastic is pretty good at breaking compatibility the > hard > > way (see in particular in the last PR the cast that have to be made to > get > > an API that works in all cases) > > - Elasticsearch is moving away from their "native" API Flink is using to > > the REST APIs so there is little common ground between pre 6 and post 6 > > even if Elasticsearch tried to get some level of compatibility in the > APIs. > > > > My fear is that by trying to kill two birds with one stone, we actually > get > > nothing done. > > > > In the hope of moving that forward I would like to propose for 1.6 a new > > Elasticsearch 6.x+ sink that would follow the design of the previous ones > > BUT only leverage the new REST API and not inherit from existing classes. > > It would really be close to what is in my previous PR: > > https://github.com/apache/flink/pull/5374 but just focus on E6+/REST and > > so > > avoid any "strange" cast. > > > > This would not fill the gap of the 5.2+ not working but at least we would > > be back on track with something for the future as REST API is where > Elastic > > is going. > > > > If people feel there is actual interest and chances this can be merged > I'll > > be working on issuing a new PR around that. > > > > Alternative is to get back working on the existing PR but it seems to be > a > > dead-end at the moment and not necessarily the best option long term as > > anyway Elasticsearch is looking into promoting the REST API. > > > > Please let me know what you think? > > > > -- > > Christophe > > >
