Hi, For reference, prior work on this: https://github.com/apache/kafka/pull/11350 https://github.com/apache/kafka/pull/11432
Best, On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lbruts...@confluent.io.invalid> wrote: > Hi Matthiases, > > I know Scala 2 fairly well, so I'd be happy to review changes that add > Scala 3 support. However, as Matthias S. said, it has to be driven by > people who use Scala day-to-day, since I believe most Kafka Streams > committers are working with Java. > > Rewriting the tests to not use EmbeddedKafkaCluster seems like a large > undertaking, so option 1 is the first thing we should explore. > > I don't have any experience with Scala 3 migration topics, but on the > Scala website it says > > The first piece of good news is that the Scala 3 compiler is able to > read the Scala 2.13 Pickle format and thus it can type check code that > depends on modules or libraries compiled with Scala 2.13. > > One notable example is the Scala 2.13 library. We have indeed decided > that the Scala 2.13 library is the official standard library for Scala 3. > So wouldn't that mean that we are safe in terms of standard library > upgrades if we use core_2.13 in the tests? > > Cheers, > Lucas > > > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org> wrote: > > > > Thanks for raising this. The `kafka-streams-scala` module seems to be an > > important feature for Kafka Streams and I am generally in favor of your > > proposal to add Scala 3 support. However, I am personally no Scala > > person and it sounds like quite some overhead. > > > > If you are willing to drive and own this initiative happy to support you > > to the extend I can. > > > > About the concrete proposal: my understanding is that :core will move > > off Scala long-term (not 100% sure what the timeline is, but new modules > > are written in Java only). Thus, down the road the compatibility issue > > would go away naturally, but it's unclear when. > > > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we > > could add support for Scala 3 now, taking a risk that it might break in > > the future assume that the migration off Scala from core is not fast > enough. > > > > For proposal (2), I don't think that it would be easily possible for > > unit/integration tests. We could fall back to system tests though, but > > they would be much more heavy weight of course. > > > > Might be good to hear from others. We might actually also want to do a > > KIP for this? > > > > > > -Matthias > > > > On 1/20/24 10:34 AM, Matthias Berndt wrote: > > > Hey there, > > > > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala library. > > > Currently, the build system is set up such that kafka-streams-scala > > > and core (i. e. kafka itself) are compiled with the same Scala > > > compiler versions. This is not an optimal situation because it means > > > that a Scala 3 release of kafka-streams-scala cannot happen > > > independently of kafka itself. I think this should be changed > > > > > > The production codebase of scala-streams-kafka actually compiles just > > > fine on Scala 3.3.1 with two lines of trivial syntax changes. The > > > problem is with the tests. These use the `EmbeddedKafkaCluster` class, > > > which means that kafka is pulled into the classpath, potentially > > > leading to binary compatibility issues. > > > I can see several approaches to fixing this: > > > > > > 1. Run the kafka-streams-scala tests using the compatible version of > > > :core if one is available. Currently, this means that everything can > > > be tested (test kafka-streams-scala_2.12 using core_2.12, > > > kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3 > > > using core_2.13, as these should be compatible), but when a new > > > scala-library version is released that is no longer compatible with > > > 2.13, we won't be able to test that. > > > 2. Rewrite the tests to run without EmbeddedKafkaCluster, instead > > > running the test cluster in a separate JVM or perhaps even a > > > container. > > > > > > I'd be willing to get my hands dirty working on this, but before I > > > start I'd like to get some feedback from the Kafka team regarding the > > > approaches outlined above. > > > > > > All the best > > > Matthias Berndt > KJosep Prat Open Source Engineering Director, aivenjosep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
