Do we all agree that we can drop Kafka 0.9/0.10 support after next LTS release (is it already known which one will be the next?) or we need to vote for this?
> On 4 Apr 2019, at 16:49, Alexey Romanenko <aromanenko....@gmail.com> wrote: > > +1 for that too. > >> On 4 Apr 2019, at 01:53, Raghu Angadi <ang...@gmail.com >> <mailto:ang...@gmail.com>> wrote: >> >> I mean, +1 for removing support for old Kafka versions after next LTS >> >> What the cut off should be for 'old' version is can be discussed then. My >> choice would be 0.11. >> Raghu. >> >> On Wed, Apr 3, 2019 at 4:36 PM Raghu Angadi <ang...@gmail.com >> <mailto:ang...@gmail.com>> wrote: >> +1 for next LTS. >> >> On Wed, Apr 3, 2019 at 2:30 PM Ismaël Mejía <ieme...@gmail.com >> <mailto:ieme...@gmail.com>> wrote: >> We should focus on the main reason to remove the Kafka 0.9 support. I >> have the impression that this is mostly to ease the maintenance, but >> from the current status (and the removal PR [1]), it does not seem >> like it is a burden to continue supporting 0.9. In any case I am +1 to >> remove the support for 0.9, but maybe it is a good idea to just wait >> until the next LTS is decided and do it just after. This way we will >> still cover existing users for some time. >> >> Creating different modules for different versions of KafkaIO does not >> make sense because it is even more complicated than just staying the >> way we are today for not much in return. We better improve the status >> quo by parametrizing our current tests to validate that KafkaIO works >> correctly with the different supported versions (so far we only test >> against version 1.0.0). I filled BEAM-7003 to track this. >> >> [1] https://github.com/apache/beam/pull/8186 >> <https://github.com/apache/beam/pull/8186> >> [2] https://issues.apache.org/jira/browse/BEAM-7003 >> <https://issues.apache.org/jira/browse/BEAM-7003> >> >> ps. Actually this discussion brings to the table the issue of >> removing/deprecated/changing supported versions on parts of the API >> marked as @Experimental. I will fork a new thread to discuss this. >> >> On Wed, Apr 3, 2019 at 6:53 PM Raghu Angadi <ang...@gmail.com >> <mailto:ang...@gmail.com>> wrote: >> > >> > >> > >> > On Wed, Apr 3, 2019 at 5:46 AM David Morávek <david.mora...@gmail.com >> > <mailto:david.mora...@gmail.com>> wrote: >> >> >> >> I'd say that APIs we use in KafkaIO are pretty much stable since 0.10 >> >> release, all reflection based compatibility adapters seem to be aimed for >> >> 0.9 release (which is 8 major releases behind current Kafka release). >> >> >> >> We may take an inspiration from Flink's kafka connector, they maintain >> >> separate maven artifact for all supported Kafka APIs. This may be the >> >> best approach as we can still share most of the codebase between >> >> versions, have compile time checks and also run tests against all of the >> >> supported versions. >> > >> > >> > From that page, Flink also moved to single Kafka connector for versions >> > 10.x and newer. Kafka itself seems to have improved compatibility between >> > client and broker versions starting 0.11. Not sure if there is any need >> > now to make multiple versions of KafkaIO versions for 0.9.x etc. Are you >> > suggesting we should? >> > >> > From Flink's page: >> > "Starting with Flink 1.7, there is a new universal Kafka connector that >> > does not track a specific Kafka major version. Rather, it tracks the >> > latest version of Kafka at the time of the Flink release. >> > >> > If your Kafka broker version is 1.0.0 or newer, you should use this Kafka >> > connector. If you use an older version of Kafka (0.11, 0.10, 0.9, or 0.8), >> > you should use the connector corresponding to the broker version." >> > >> > >> >> >> >> >> >> I'm not really comfortable with reflection based adapters as they seem >> >> fragile and don't provide compile time checks. >> >> >> >> On Tue, Apr 2, 2019 at 11:27 PM Austin Bennett >> >> <whatwouldausti...@gmail.com <mailto:whatwouldausti...@gmail.com>> wrote: >> >>> >> >>> I withdraw my concern -- checked on info on the cluster I will >> >>> eventually access. It is on 0.8, so I was speaking too soon. Can't >> >>> speak to rest of user base. >> >>> >> >>> On Tue, Apr 2, 2019 at 11:03 AM Raghu Angadi <ang...@gmail.com >> >>> <mailto:ang...@gmail.com>> wrote: >> >>>> >> >>>> Thanks to David Morávek for pointing out possible improvement to >> >>>> KafkaIO for dropping support for 0.9 since it avoids having a second >> >>>> consumer just to fetch latest offsets for backlog. >> >>>> >> >>>> Ideally we should be dropping 0.9 support for next major release, in >> >>>> fact better to drop versions before 0.10.1 at the same time. This would >> >>>> further reduce reflection based calls for supporting multiple versions. >> >>>> If the users still on 0.9 could stay on current stable release of Beam, >> >>>> dropping would not affect them. Otherwise, it would be good to hear >> >>>> from them about how long we need to keep support for old versions. >> >>>> >> >>>> I don't think it is good idea to have multiple forks of KafkaIO in the >> >>>> same repo. If we do go that route, we should fork the entire kafka >> >>>> directory and rename the main class KafkaIO_Unmaintained :). >> >>>> >> >>>> IMHO, so far, additional complexity for supporting these versions is >> >>>> not that bad. Most of it is isolated to ConsumerSpEL.java & >> >>>> ProducerSpEL.java. >> >>>> My first preference is dropping support for deprecated versions (and a >> >>>> deprecate a few more versions, may be till the version that added >> >>>> transactions around 0.11.x I think). >> >>>> >> >>>> I haven't looked into what's new in Kafka 2.x. Are there any features >> >>>> that KafkaIO should take advantage of? I have not noticed our existing >> >>>> code breaking. We should certainly certainly support latest releases of >> >>>> Kafka. >> >>>> >> >>>> Raghu. >> >>>> >> >>>> On Tue, Apr 2, 2019 at 10:27 AM Mingmin Xu <mingm...@gmail.com >> >>>> <mailto:mingm...@gmail.com>> wrote: >> >>>>> >> >>>>> >> >>>>> We're still using Kafka 0.10 a lot, similar as 0.9 IMO. To expand >> >>>>> multiple versions in KafkaIO is quite complex now, and it confuses >> >>>>> users which is supported / which is not. I would prefer to support >> >>>>> Kafka 2.0+ only in the latest version. For old versions, there're some >> >>>>> options: >> >>>>> 1). document Kafka-Beam support versions, like what we do in >> >>>>> FlinkRunner; >> >>>>> 2). maintain separated KafkaIOs for old versions; >> >>>>> >> >>>>> 1) would be easy to maintain, and I assume there should be no issue to >> >>>>> use Beam-Core 3.0 together with KafkaIO 2.0. >> >>>>> >> >>>>> Any thoughts? >> >>>>> >> >>>>> Mingmin >> >>>>> >> >>>>> On Tue, Apr 2, 2019 at 9:56 AM Reuven Lax <re...@google.com >> >>>>> <mailto:re...@google.com>> wrote: >> >>>>>> >> >>>>>> KafkaIO is marked as Experimental, and the comment already warns that >> >>>>>> 0.9 support might be removed. I think that if users still rely on >> >>>>>> Kafka 0.9 we should leave a fork (renamed) of the IO in the tree for >> >>>>>> 0.9, but we can definitely remove 0.9 support from the main IO if we >> >>>>>> want, especially if it's complicated changes to that IO. If we do >> >>>>>> though, we should fail with a clear error message telling users to >> >>>>>> use the Kafka 0.9 IO. >> >>>>>> >> >>>>>> On Tue, Apr 2, 2019 at 9:34 AM Alexey Romanenko >> >>>>>> <aromanenko....@gmail.com <mailto:aromanenko....@gmail.com>> wrote: >> >>>>>>> >> >>>>>>> > How are multiple versions of Kafka supported? Are they all in one >> >>>>>>> > client, or is there a case for forks like ElasticSearchIO? >> >>>>>>> >> >>>>>>> They are supported in one client but we have additional >> >>>>>>> “ConsumerSpEL” adapter which unifies interface difference among >> >>>>>>> different Kafka client versions (mostly to support old ones >> >>>>>>> 0.9-0.10.0). >> >>>>>>> >> >>>>>>> On the other hand, we warn user in Javadoc of KafkaIO (which is >> >>>>>>> Unstable, btw) by the following: >> >>>>>>> “KafkaIO relies on kafka-clients for all its interactions with the >> >>>>>>> Kafka cluster.kafka-clients versions 0.10.1 and newer are supported >> >>>>>>> at runtime. The older versions 0.9.x - 0.10.0.0 are also supported, >> >>>>>>> but are deprecated and likely be removed in near future.” >> >>>>>>> >> >>>>>>> Despite the fact that, personally, I’d prefer to have only one >> >>>>>>> unified client interface but, since people still use Beam with old >> >>>>>>> Kafka instances, we, likely, should stick with it till Beam 3.0. >> >>>>>>> >> >>>>>>> WDYT? >> >>>>>>> >> >>>>>>> On 2 Apr 2019, at 02:27, Austin Bennett <whatwouldausti...@gmail.com >> >>>>>>> <mailto:whatwouldausti...@gmail.com>> wrote: >> >>>>>>> >> >>>>>>> FWIW -- >> >>>>>>> >> >>>>>>> On my (desired, not explicitly job-function) roadmap is to tap into >> >>>>>>> a bunch of our corporate Kafka queues to ingest that data to places >> >>>>>>> I can use. Those are 'stuck' 0.9, with no upgrade in sight (am told >> >>>>>>> the upgrade path isn't trivial, is very critical flows, and they are >> >>>>>>> scared for it to break, so it just sits behind firewalls, etc). >> >>>>>>> But, I wouldn't begin that for probably at least another quarter. >> >>>>>>> >> >>>>>>> I don't contribute to nor understand the burden of maintaining the >> >>>>>>> support for the older version, so can't reasonably lobby for that >> >>>>>>> continued pain. >> >>>>>>> >> >>>>>>> Anecdotally, this could be a place many enterprises are at (though I >> >>>>>>> also wonder whether many of the people that would be 'stuck' on such >> >>>>>>> versions would also have Beam on their current radar). >> >>>>>>> >> >>>>>>> >> >>>>>>> On Mon, Apr 1, 2019 at 2:29 PM Kenneth Knowles <k...@apache.org >> >>>>>>> <mailto:k...@apache.org>> wrote: >> >>>>>>>> >> >>>>>>>> This could be a backward-incompatible change, though that notion >> >>>>>>>> has many interpretations. What matters is user pain. Technically if >> >>>>>>>> we don't break the core SDK, users should be able to use Java SDK >> >>>>>>>> >=2.11.0 with KafkaIO 2.11.0 forever. >> >>>>>>>> >> >>>>>>>> How are multiple versions of Kafka supported? Are they all in one >> >>>>>>>> client, or is there a case for forks like ElasticSearchIO? >> >>>>>>>> >> >>>>>>>> Kenn >> >>>>>>>> >> >>>>>>>> On Mon, Apr 1, 2019 at 10:37 AM Jean-Baptiste Onofré >> >>>>>>>> <j...@nanthrax.net <mailto:j...@nanthrax.net>> wrote: >> >>>>>>>>> >> >>>>>>>>> +1 to remove 0.9 support. >> >>>>>>>>> >> >>>>>>>>> I think it's more interesting to test and verify Kafka 2.2.0 than >> >>>>>>>>> 0.9 ;) >> >>>>>>>>> >> >>>>>>>>> Regards >> >>>>>>>>> JB >> >>>>>>>>> >> >>>>>>>>> On 01/04/2019 19:36, David Morávek wrote: >> >>>>>>>>> > Hello, >> >>>>>>>>> > >> >>>>>>>>> > is there still a reason to keep Kafka 0.9 support? This >> >>>>>>>>> > unfortunately >> >>>>>>>>> > adds lot of complexity to KafkaIO implementation. >> >>>>>>>>> > >> >>>>>>>>> > Kafka 0.9 was released on Nov 2015. >> >>>>>>>>> > >> >>>>>>>>> > My first shot on removing Kafka 0.9 support would remove second >> >>>>>>>>> > consumer, which is used for fetching offsets. >> >>>>>>>>> > >> >>>>>>>>> > WDYT? Is this support worth keeping? >> >>>>>>>>> > >> >>>>>>>>> > https://github.com/apache/beam/pull/8186 >> >>>>>>>>> > <https://github.com/apache/beam/pull/8186> >> >>>>>>>>> > >> >>>>>>>>> > D. >> >>>>>>>>> >> >>>>>>>>> -- >> >>>>>>>>> Jean-Baptiste Onofré >> >>>>>>>>> jbono...@apache.org <mailto:jbono...@apache.org> >> >>>>>>>>> http://blog.nanthrax.net <http://blog.nanthrax.net/> >> >>>>>>>>> Talend - http://www.talend.com <http://www.talend.com/> >> >>>>>>> >> >>>>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> ---- >> >>>>> Mingmin >
