Hi David, that's a discussion we had while ago. I was more in favor of having module per main backend version:
- kafka 0.9 - kafka 1.x - kafka 2.x for instance. It's what we do in some Apache projects (like Camel for instance). As a best effort, we can try to maintain a unique module for a backend, but it makes sense to dedicate a module to avoid to be blocked. Regards JB On 03/04/2019 14:46, David Morávek wrote: > I'd say that APIs we use in KafkaIO are pretty much stable since 0.10 > release, all reflection based compatibility adapters seem to be aimed > for 0.9 release (which is 8 major releases behind current Kafka release). > > We may take an inspiration from Flink's kafka connector > <https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html>, > they maintain separate maven artifact for all supported Kafka APIs. This > may be the best approach as we can still share most of the codebase > between versions, have compile time checks and also run tests against > all of the supported versions. > > I'm not really comfortable with reflection based adapters > <https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java> > as they seem fragile and don't provide compile time checks. > > On Tue, Apr 2, 2019 at 11:27 PM Austin Bennett > <[email protected] <mailto:[email protected]>> wrote: > > I withdraw my concern -- checked on info on the cluster I will > eventually access. It is on 0.8, so I was speaking too soon. Can't > speak to rest of user base. > > On Tue, Apr 2, 2019 at 11:03 AM Raghu Angadi <[email protected] > <mailto:[email protected]>> wrote: > > Thanks to David Morávek for pointing out possible improvement to > KafkaIO for dropping support for 0.9 since it avoids having a > second consumer just to fetch latest offsets for backlog. > > Ideally we should be dropping 0.9 support for next major > release, in fact better to drop versions before 0.10.1 at the > same time. This would further reduce reflection based calls for > supporting multiple versions. If the users still on 0.9 could > stay on current stable release of Beam, dropping would not > affect them. Otherwise, it would be good to hear from them about > how long we need to keep support for old versions. > > I don't think it is good idea to have multiple forks of KafkaIO > in the same repo. If we do go that route, we should fork the > entire kafka directory and rename the main class > KafkaIO_Unmaintained :). > > IMHO, so far, additional complexity for supporting these > versions is not that bad. Most of it is isolated to > ConsumerSpEL.java & ProducerSpEL.java. > My first preference is dropping support for deprecated versions > (and a deprecate a few more versions, may be till the version > that added transactions around 0.11.x I think). > > I haven't looked into what's new in Kafka 2.x. Are there any > features that KafkaIO should take advantage of? I have not > noticed our existing code breaking. We should certainly > certainly support latest releases of Kafka. > > Raghu. > > On Tue, Apr 2, 2019 at 10:27 AM Mingmin Xu <[email protected] > <mailto:[email protected]>> wrote: > > > We're still using Kafka 0.10 a lot, similar as 0.9 IMO. To > expand multiple versions in KafkaIO is quite complex now, > and it confuses users which is supported / which is not. I > would prefer to support Kafka 2.0+ only in the latest > version. For old versions, there're some options: > 1). document Kafka-Beam support versions, like what we do in > FlinkRunner; > 2). maintain separated KafkaIOs for old versions; > > 1) would be easy to maintain, and I assume there should be > no issue to use Beam-Core 3.0 together with KafkaIO 2.0. > > Any thoughts? > > Mingmin > > On Tue, Apr 2, 2019 at 9:56 AM Reuven Lax <[email protected] > <mailto:[email protected]>> wrote: > > KafkaIO is marked as Experimental, and the comment > already warns that 0.9 support might be removed. I think > that if users still rely on Kafka 0.9 we should leave a > fork (renamed) of the IO in the tree for 0.9, but we can > definitely remove 0.9 support from the main IO if we > want, especially if it's complicated changes to that IO. > If we do though, we should fail with a clear error > message telling users to use the Kafka 0.9 IO. > > On Tue, Apr 2, 2019 at 9:34 AM Alexey Romanenko > <[email protected] > <mailto:[email protected]>> wrote: > > > How are multiple versions of Kafka supported? Are > they all in one client, or is there a case for forks > like ElasticSearchIO? > > They are supported in one client but we have > additional “ConsumerSpEL” adapter which unifies > interface difference among different Kafka client > versions (mostly to support old ones 0.9-0.10.0). > > On the other hand, we warn user in Javadoc of > KafkaIO (which is Unstable, btw) by the following: > /“KafkaIO relies on kafka-clients for all its > interactions with the Kafka > cluster.//kafka-clients versions 0.10.1 and newer > are supported at runtime. The older versions > 0.9.x //- 0.10.0.0 are also supported, but are > deprecated and likely be removed in near future.”/ > / > / > Despite the fact that, personally, I’d prefer to > have only one unified client interface but, since > people still use Beam with old Kafka instances, we, > likely, should stick with it till Beam 3.0. > > WDYT? > >> On 2 Apr 2019, at 02:27, Austin Bennett >> <[email protected] >> <mailto:[email protected]>> wrote: >> >> FWIW -- >> >> On my (desired, not explicitly job-function) >> roadmap is to tap into a bunch of our corporate >> Kafka queues to ingest that data to places I can >> use. Those are 'stuck' 0.9, with no upgrade in >> sight (am told the upgrade path isn't trivial, is >> very critical flows, and they are scared for it to >> break, so it just sits behind firewalls, etc). >> But, I wouldn't begin that for probably at least >> another quarter. >> >> I don't contribute to nor understand the burden of >> maintaining the support for the older version, so >> can't reasonably lobby for that continued pain. >> >> Anecdotally, this could be a place many >> enterprises are at (though I also wonder whether >> many of the people that would be 'stuck' on such >> versions would also have Beam on their current >> radar). >> >> >> On Mon, Apr 1, 2019 at 2:29 PM Kenneth Knowles >> <[email protected] <mailto:[email protected]>> wrote: >> >> This could be a backward-incompatible change, >> though that notion has many interpretations. >> What matters is user pain. Technically if we >> don't break the core SDK, users should be able >> to use Java SDK >=2.11.0 with KafkaIO 2.11.0 >> forever. >> >> How are multiple versions of Kafka supported? >> Are they all in one client, or is there a case >> for forks like ElasticSearchIO? >> >> Kenn >> >> On Mon, Apr 1, 2019 at 10:37 AM Jean-Baptiste >> Onofré <[email protected] >> <mailto:[email protected]>> wrote: >> >> +1 to remove 0.9 support. >> >> I think it's more interesting to test and >> verify Kafka 2.2.0 than 0.9 ;) >> >> Regards >> JB >> >> On 01/04/2019 19:36, David Morávek wrote: >> > Hello, >> > >> > is there still a reason to keep Kafka >> 0.9 support? This unfortunately >> > adds lot of complexity to KafkaIO >> implementation. >> > >> > Kafka 0.9 was released on Nov 2015. >> > >> > My first shot on removing Kafka 0.9 >> support would remove second >> > consumer, which is used for fetching >> offsets. >> > >> > WDYT? Is this support worth keeping? >> > >> > https://github.com/apache/beam/pull/8186 >> > >> > D. >> >> -- >> Jean-Baptiste Onofré >> [email protected] >> <mailto:[email protected]> >> http://blog.nanthrax.net >> <http://blog.nanthrax.net/> >> Talend - http://www.talend.com >> <http://www.talend.com/> >> > > > > -- > ---- > Mingmin > -- Jean-Baptiste Onofré [email protected] http://blog.nanthrax.net Talend - http://www.talend.com
