Hi there, I just wanted to bump this thread as there has been a great development. Gradle 7.3 is now released and it comes with Scala 3 support (release notes <https://docs.gradle.org/7.3/release-notes.html>). This means that one of the roadblocks to migrate Kafka to Scala 3 has been cleared. Shall I write a KIP to decide if we want to migrate to Scala 3 right away or wait until Kafka 4.0 in which Scala 2.12 will be removed?
I also had a favour to ask, could someone take a look at https://github.com/apache/kafka/pull/11432? The purpose of the PR is to improve some Scala code to make it easy to migrate in the future, the PR has been sitting there for several weeks now. I would highly appreciate some reviews in there so we can move it forward before more merge conflicts appear. Thanks a lot! On Tue, Nov 2, 2021 at 9:06 AM Josep Prat <josep.p...@aiven.io> wrote: > Hi Colin, > > We can take 2 different paths, both of them need a similar effort. > — Offer Scala 3 along with Scala 2.12 and 2.13 > — Offer Scala 3 only once Scala 2.12 is removed > > On the draft PR I shared in the initial email ( > https://github.com/apache/kafka/pull/11350), I used the first approach, > and it didn't require any additional work. This means we can decide to take > any of the 2 approaches freely. Offering Scala 3 only when 2.13 is removed > means to wait for a bit, but it's perfectly doable. > My personal.opinion is that, given Scala 3 is a major release (epoch > according to Scala version schema), we might want to build in Scala 3 > sooner rather than later. We could label this as experimental Scala 3 > support. > However, until Gradle 7.3 is released, we can't proceed any further with > building with Scala 3. Currently, Gradle 7.3 is under RC3, so I expect it > to be available soon. > > Regarding the changes I propose to add right now as I mentioned in the > first email, you can see exactly what I meant in this PR: > https://github.com/apache/kafka/pull/11432. Sometimes, Kafka uses some > syntax or approaches that are deprecated and they are removed in Scala 3. > This PR goes over those and uses the "accepted" syntax. Feel free to take > a look and review it. > > Thanks for your feedback Colin! > > Best, > ——— > Josep Prat > > Aiven Deutschland GmbH > > Immanuelkirchstraße 26, 10405 Berlin > > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > m: +491715557497 > > w: aiven.io > > e: josep.p...@aiven.io > > On Mon, Nov 1, 2021, 23:45 Colin McCabe <cmcc...@apache.org> wrote: > >> Thanks for looking at this, Josep... I guess the way I always imagined >> this happening was that a Scala 3 release would become one of our two >> supported Scala versions. So instead of 2.12 and 2.13, we'd have 2.13 and >> 3.x, for some X. Do you think that approach would be practical? >> >> best, >> Colin >> >> >> On Fri, Oct 8, 2021, at 09:05, Ismael Juma wrote: >> > Yeah, changes that are good generally can be submitted any time. >> > >> > Ismael >> > >> > On Fri, Oct 8, 2021 at 7:35 AM Josep Prat <josep.p...@aiven.io.invalid> >> > wrote: >> > >> >> Hi Ismael, >> >> >> >> Thanks for the reply. I don't think the demand is high at the moment, >> but >> >> it will increase overtime. >> >> What do you think about me submitting a PR now with only the changes I >> >> mention in group a)? They are small changes like dropping extra >> >> parenthesis, or stopping shadowing names. The resulting code would be >> >> cleaner Scala that can cross compile. >> >> >> >> Thanks again, >> >> ——— >> >> Josep Prat >> >> >> >> Aiven Deutschland GmbH >> >> >> >> Immanuelkirchstraße 26, 10405 Berlin >> >> >> >> Amtsgericht Charlottenburg, HRB 209739 B >> >> >> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen >> >> >> >> m: +491715557497 >> >> >> >> w: aiven.io >> >> >> >> e: josep.p...@aiven.io >> >> >> >> On Fri, Oct 8, 2021, 16:22 Ismael Juma <ism...@juma.me.uk> wrote: >> >> >> >> > Hi Josep, >> >> > >> >> > Thanks for looking into this. As you said, it seems like there are a >> >> number >> >> > of rough edges still. Do we have many users asking for this right >> now? If >> >> > not, it may make sense to wait for things to get a bit more mature >> before >> >> > adding the burden of a third Scala version. Ideally, we would drop >> Scala >> >> > 2.12 before we add Scala 3 support. >> >> > >> >> > Ismael >> >> > >> >> > On Fri, Oct 8, 2021 at 6:07 AM Josep Prat >> <josep.p...@aiven.io.invalid> >> >> > wrote: >> >> > >> >> > > Hi there, >> >> > > For the last month I was actively working on migrating Apache >> Kafka to >> >> > > Scala 3. For what I could see, Apache Kafka is the biggest mixed >> >> > Java-Scala >> >> > > project to attempt a migration to Scala 3. This means, a lot of >> >> > regressions >> >> > > in regards to Java and Scala interoperability at bytecode were >> found >> >> > within >> >> > > this task. You can take a look at the PR here: >> >> > > https://github.com/apache/kafka/pull/11350 >> >> > > Currently, the status of the migration is as follows: >> >> > > >> >> > > - All tests pass in Scala 3 except one ( >> >> > > >> >> > > >> >> > >> >> >> KafkaApis#getAllTopicMetadataShouldNotCreateTopicOrReturnUnknownTopicPartition). >> >> > > Root cause for this is this bug with no easy workaround: Type >> erased >> >> > for >> >> > > by-name parameters lampepfl/dotty#13638 >> >> > > <https://github.com/lampepfl/dotty/issues/13638> >> >> > > - All tests pass in Scala 2 >> >> > > - It still uses a snapshot build of Gradle with support for >> Scala 3 >> >> > > - Spotbugs detected some new errors when the code was compiled >> with >> >> > > Scala 3. Some are simply false positives, but a small subset >> seem >> >> like >> >> > > they >> >> > > might be legit bugs in Scala 3. I still need to come up with a >> >> > minimized >> >> > > reproducer for those cases. >> >> > > >> >> > > As one can see, most of the changes are arguably low-level and no >> big >> >> > > rewrite was needed. The biggest refactoring was splitting several >> >> classes >> >> > > into their own file. >> >> > > >> >> > > Changes made in this PR could be categorized in 3 different groups: >> >> > > a) Needed changes because of changes in syntax or compiler being >> >> > currently >> >> > > more strict >> >> > > b) Changes where a workaround is present but they are (or will be) >> >> fixed >> >> > in >> >> > > upcoming Scala versions >> >> > > c) Changes in the build file >> >> > > >> >> > > In order to have a smooth and manageable migration to Scala 3, I >> >> propose >> >> > > perform the migration in several steps: >> >> > > >> >> > > - Firstly apply all changes that would fall into category *a)* >> >> > > - Once a new Scala version that incorporates the mentioned >> fixes is >> >> > > released, revisit point changes in point *b)* >> >> > > - Once Type erased for by-name parameters lampepfl/dotty#13638 >> >> > > <https://github.com/lampepfl/dotty/issues/13638> is resolved >> or a >> >> > > workaround is found and Gradle with Scala 3 support is released >> we >> >> > could >> >> > > tackle the final step and incorporate changes mentioned in point >> >> *c)* >> >> > > - After this, we would run the Scala Migration Tool to >> automatically >> >> > > rewrite the code that uses removed syntax or language >> constructs. >> >> (See >> >> > > >> >> > > >> >> > >> >> >> https://docs.scala-lang.org/scala3/guides/migration/tooling-migration-mode.html >> >> > > ) >> >> > > >> >> > > >> >> > > What do you think about this approach? Please feel free to share >> your >> >> > > opinion on the proposed plan. >> >> > > Also, I'm not sure if this should be written in a KIP form or not, >> no >> >> > > public interfaces needed to be modified. >> >> > > >> >> > > >> >> > > For extra information, these are the bugs reported to Scala 3 so >> far: >> >> > > >> >> > > - https://github.com/lampepfl/dotty/issues/13549 >> >> > > - https://github.com/lampepfl/dotty/issues/13572 >> >> > > - https://github.com/lampepfl/dotty/issues/13630 >> >> > > - https://github.com/lampepfl/dotty/issues/13638 >> >> > > - https://github.com/lampepfl/dotty/issues/13645 >> >> > > >> >> > > >> >> > > Thank you in advance! >> >> > > -- >> >> > > >> >> > > Josep Prat >> >> > > >> >> > > *Aiven Deutschland GmbH* >> >> > > >> >> > > Immanuelkirchstraße 26, 10405 Berlin >> >> > > >> >> > > Amtsgericht Charlottenburg, HRB 209739 B >> >> > > >> >> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen >> >> > > >> >> > > *m:* +491715557497 >> >> > > >> >> > > *w:* aiven.io >> >> > > >> >> > > *e:* josep.p...@aiven.io >> >> > > >> >> > >> >> >> > -- Josep Prat *Aiven Deutschland GmbH* Immanuelkirchstraße 26, 10405 Berlin Amtsgericht Charlottenburg, HRB 209739 B Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen *m:* +491715557497 *w:* aiven.io *e:* josep.p...@aiven.io
