Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK

2020-11-04 Thread Alexey Romanenko
> On 30 Oct 2020, at 19:33, Piotr Szuberski wrote: > > Oh I see, I thought that our tests are testing different Kafka versions right > now. There is a test suite for 2.1.0 version in addition to 1.0.0 if I > understand that gradle.build file correctly. Yes and iirc, it was added to test Kaf

Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK

2020-10-30 Thread Piotr Szuberski
Oh I see, I thought that our tests are testing different Kafka versions right now. There is a test suite for 2.1.0 version in addition to 1.0.0 if I understand that gradle.build file correctly. I don't mean to drop the support to earlier Kafka versions. I just mean to set the recent versions as

Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK

2020-10-30 Thread Alexey Romanenko
We have a Jira task about testing KafkaIO against different Kafka versions [1]. Once it will be implemented, then I think it should solve a potential problem with new Kafka client versions. In the same time, imho, until there are Beam users, that use old Kafka client versions, we need to suppo

Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK

2020-10-30 Thread Piotr Szuberski
In my opinion it would be good to keep Beam's dependencies as close to the recent stable versions as possible and, if needed, keep the support for earlier versions. For now we keep the old dependency as the base and test whether it works for some newer versions. That way we may always ignore th

Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK

2020-10-28 Thread Alexey Romanenko
*raising this question, of course > On 28 Oct 2020, at 18:06, Alexey Romanenko wrote: > > tasing this question

Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK

2020-10-28 Thread Alexey Romanenko
Piotr, thank you for tasing this question. Let me ask some questions before. What will give us this dependencies update? What are the pros and cons? Can users use recent versions of Kafka client with current implementation based on ConsumerSpEL class? > On 22 Oct 2020, at 10:47, Piotr Szubersk

[DISCUSS] Update Kafka dependencies in Beam Java SDK

2020-10-22 Thread Piotr Szuberski
Should we update Kafka dependencies to the recent ones (Kafka clients to 2.6.0 and Kafka_2.11 to 2.4.1)? What would have to be done to keep the backwards compatibility and which previous versions would we want to support? Kafka's backward compatibility is quite good so maybe there wouldn't be