Hi Colin, I see that you made 3.9 the default docs version in Kafka-Site. However, as your doc changes were pushed before the 3.8.1 release, now the upgrade.html page doesn't mention 3.8.1 at all. I'll update the site.
On Wed, Nov 6, 2024 at 4:48 PM Viktor Somogyi-Vass <viktor.somo...@cloudera.com.invalid> wrote: > Ok, sorry for the noise. It seems like my setup was incorrectly configured, > one controller had an incorrect voters config and that caused the issue. > Retested it and it works now. > But given this, I might as well just give a +1. :) > > On Wed, Nov 6, 2024 at 2:53 PM Viktor Somogyi-Vass < > viktor.somo...@cloudera.com> wrote: > > > Hey All, > > > > It seems like I ran into an issue. While trying to do the migration (3.9 > > ZK -> 3.9 KRaft), at the very last step the whole cluster (except the > > controller I just restarted) fails with a fatal error where a controller > (I > > think) requests a truncation that is below the HWM. > > I tried to capture everything but with the configs and the steps I > > described it can be reproduced every time. > > > > I started looking into it, but since I'm completely unfamiliar with many > > parts of KRaft, I'd be happy if someone else more familiar with it could > > look at it. Especially if I didn't follow the migration procedure well > (but > > I think I did so). > > All the info captured here: > > https://issues.apache.org/jira/browse/KAFKA-17950 > > > > Viktor > > > > > > On Fri, Nov 1, 2024 at 11:00 PM Colin McCabe <cmcc...@apache.org> wrote: > > > >> Argh. That should read "5 +1 votes". :) > >> > >> Thanks, everyone. > >> Colin > >> > >> > >> On Fri, Nov 1, 2024, at 14:58, Colin McCabe wrote: > >> > Hi all, > >> > > >> > This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes. > >> > > >> > +1 votes > >> > PMC Members: > >> > * Justine Olshan > >> > * Bill Bejeck > >> > * David Arthur > >> > * Luke Chen > >> > > >> > Community: > >> > * Federico Valeri > >> > > >> > 0 votes > >> > * No votes > >> > > >> > -1 votes > >> > * No votes > >> > > >> > Vote thread: > >> > https://lists.apache.org/thread/og8l24z8oj87x0zsoqddfxo54qsj4d1o > >> > > >> > I'll continue with the release process and the release announcement > >> > will follow in the next few days. > >> > > >> > best, > >> > Colin > >> > > >> > > >> > On Thu, Oct 31, 2024, at 21:03, Luke Chen wrote: > >> >> Hi Colin, > >> >> > >> >> Thanks for the update. > >> >> I've ran a sanity check as Mickael suggested for the maven artifacts > >> with > >> >> the one in https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/. > >> >> I used jarcomp < > >> https://activityworkshop.net/software/jarcomp/index.html> > >> >> to verify: > >> >> 1. kafka_2.13-3.9.0.jar > >> >> 2. kafka-clients-3.9.0.jar > >> >> 3. kafka-metadata-3.9.0.jar > >> >> > >> >> I confirmed all the class files in the jars have the same content. > >> >> > >> >> So +1 (binding) from me. > >> >> > >> >> Thank you. > >> >> Luke > >> >> > >> >> On Fri, Nov 1, 2024 at 1:08 AM Mickael Maison < > >> mickael.mai...@gmail.com> > >> >> wrote: > >> >> > >> >>> Hi, > >> >>> > >> >>> Can we add sanity checks to ensure all artifacts are present when > >> >>> running the release? > >> >>> Also should we use a different folder for each RC? This would > >> >>> eliminate the risk of artifacts from previous RCs hiding an issue. > >> >>> Luke spotted the old artifacts this time but this could have been > >> >>> missed. > >> >>> > >> >>> Thanks, > >> >>> Mickael > >> >>> > >> >>> On Thu, Oct 31, 2024 at 5:55 PM Colin McCabe <cmcc...@apache.org> > >> wrote: > >> >>> > > >> >>> > Hi David, > >> >>> > > >> >>> > Thanks for checking. I took a look and yes, I did forget to upload > >> the > >> >>> Scala 2.12 artifacts previously. The script runs gradlewAll, but > when > >> I was > >> >>> trying to do the steps manually, I missed that and just ran gradlew, > >> which > >> >>> only did Scala 2.13. > >> >>> > > >> >>> > This should be fixed now! > >> >>> > > >> >>> > The scala 2.12 artifacts are here: > >> >>> > > >> >>> > > >> https://repository.apache.org/content/repositories/orgapachekafka-1827/ > >> >>> > > >> >>> > and also here: > >> >>> > > >> >>> > > >> >>> > >> > https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.12/3.9.0/ > >> >>> > > >> >>> > I wonder if we should change the release.py script to just run the > >> >>> gradlew steps manually. The gradlewAll script seems a bit confusing > >> (or > >> >>> maybe that's just me?) > >> >>> > > >> >>> > best, > >> >>> > Colin > >> >>> > > >> >>> > > >> >>> > On Wed, Oct 30, 2024, at 11:02, David Arthur wrote: > >> >>> > > Should there be Scala 2.12 artifacts in the Maven staging? > >> >>> > > > >> >>> > > I see these for 2.13: > >> >>> > > * > >> >>> > > > >> >>> > >> > https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.13/3.9.0/ > >> >>> > > * > >> >>> > > > >> >>> > >> > https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka-streams-scala_2.13/3.9.0/ > >> >>> > > > >> >>> > > but I don't see 2.12 for 3.9.0 release. > >> >>> > > > >> >>> > > I can confirm that the other artifacts (such as kafka-clients) > do > >> have > >> >>> > > today's timestamp for the 3.9.0 artifact. > >> >>> > > > >> >>> > > -David > >> >>> > > > >> >>> > > On Wed, Oct 30, 2024 at 12:36 PM Colin McCabe < > cmcc...@apache.org > >> > > >> >>> wrote: > >> >>> > > > >> >>> > >> Thanks for checking, Luke. I did actually visit > >> >>> > >> https://repository.apache.org/#stagingRepositories and hit > >> close on > >> >>> the > >> >>> > >> artifacts as the release.py script says to do. I don't know why > >> they > >> >>> ended > >> >>> > >> up not showing up. > >> >>> > >> > >> >>> > >> Today I dropped all the maven artifacts for previous 3.9 RCs, > >> >>> switched to > >> >>> > >> java 8, and re-ran "gradlew publish" and "mvn deploy > >> -Pgpg-signing". > >> >>> Then I > >> >>> > >> closed the two artifacts that were uploaded. You can see them > >> here: > >> >>> > >> > >> >>> > >> > >> >>> > >> https://repository.apache.org/content/repositories/orgapachekafka-1825/ > >> >>> > >> > >> >>> > >> https://repository.apache.org/content/repositories/orgapachekafka-1826/ > >> >>> > >> > >> >>> > >> Hopefully these should now show up in Maven as expected. > >> >>> > >> > >> >>> > >> best, > >> >>> > >> Colin > >> >>> > >> > >> >>> > >> > >> >>> > >> On Tue, Oct 29, 2024, at 23:51, Luke Chen wrote: > >> >>> > >> > Hi Colin, > >> >>> > >> > > >> >>> > >> > I was trying to test the RC, but I found the artifacts in > Maven > >> >>> artifacts > >> >>> > >> > to be voted upon are not up-to-date. > >> >>> > >> > Not only the "Last Modified" time in the 3.9.0 artifacts are > on > >> >>> Oct. 10, > >> >>> > >> > but also the source code in the artifacts didn't include the > >> latest > >> >>> > >> commits > >> >>> > >> > in RC5 here < > >> https://github.com/apache/kafka/commits/3.9.0-rc5/>. > >> >>> > >> > > >> >>> > >> > Could you help verify it? > >> >>> > >> > > >> >>> > >> > But I confirmed the release artifacts in " > >> >>> > >> > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/"; is > >> >>> up-to-date. > >> >>> > >> > I did: > >> >>> > >> > - checked the checksums and signatures > >> >>> > >> > - ran upgrade from v3.8.1 to v3.9.0 > >> >>> > >> > - ran quickstart with the 2.13 binaries > >> >>> > >> > - verify tiered storage and KIP-950 (dynamically disablement) > >> >>> > >> > > >> >>> > >> > I'll be +1 after the maven artifacts are updated. > >> >>> > >> > > >> >>> > >> > Thank you for running the release. > >> >>> > >> > Luke > >> >>> > >> > > >> >>> > >> > > >> >>> > >> > On Wed, Oct 30, 2024 at 5:40 AM Justine Olshan > >> >>> > >> <jols...@confluent.io.invalid> > >> >>> > >> > wrote: > >> >>> > >> > > >> >>> > >> >> Hey Colin, > >> >>> > >> >> > >> >>> > >> >> Thanks for the new RC. I've tested the --feature > functionality > >> >>> that was > >> >>> > >> >> missing before. Seems to be working as expected now. > >> >>> > >> >> As a side note, I was looking into how to configure kip-853 > >> and it > >> >>> took > >> >>> > >> me > >> >>> > >> >> a moment to find the instructions in the documentation, I > >> wonder > >> >>> if it > >> >>> > >> >> could be included in the notable changes section of the > >> upgrade > >> >>> notes. > >> >>> > >> Not > >> >>> > >> >> a blocker as I believe we can still update documentation. > >> >>> > >> >> > >> >>> > >> >> I did some other spot checks on the rest of the release. +1 > >> >>> (binding) > >> >>> > >> from > >> >>> > >> >> me > >> >>> > >> >> > >> >>> > >> >> Justine > >> >>> > >> >> > >> >>> > >> >> On Tue, Oct 29, 2024 at 12:45 PM Colin McCabe < > >> cmcc...@apache.org> > >> >>> > >> wrote: > >> >>> > >> >> > >> >>> > >> >> > Thanks, Anton. And thanks Chia-Ping Tsai for taking a look > >> at > >> >>> how we > >> >>> > >> can > >> >>> > >> >> > improve the docs here... > >> >>> > >> >> > > >> >>> > >> >> > best, > >> >>> > >> >> > Colin > >> >>> > >> >> > > >> >>> > >> >> > > >> >>> > >> >> > On Tue, Oct 29, 2024, at 02:39, Anton Agestam wrote: > >> >>> > >> >> > > Hi Chia-Ping, > >> >>> > >> >> > > > >> >>> > >> >> > > Thanks for pointing those two fields out. I retract my > -1. > >> >>> > >> >> > > > >> >>> > >> >> > > Cheers, > >> >>> > >> >> > > Anton > >> >>> > >> >> > > > >> >>> > >> >> > > Den sön 27 okt. 2024 kl 17:40 skrev Chia-Ping Tsai < > >> >>> > >> >> chia7...@apache.org > >> >>> > >> >> > >: > >> >>> > >> >> > > > >> >>> > >> >> > >> hi Anton > >> >>> > >> >> > >> > >> >>> > >> >> > >> Thanks for sharing your insights on Kafka > >> serialization—it’s > >> >>> really > >> >>> > >> >> cool > >> >>> > >> >> > >> and interesting to me. Additionally, you inspired me to > >> file > >> >>> a JIRA > >> >>> > >> >> > issue > >> >>> > >> >> > >> (KAFKA-17882) to improve the documentation. > >> >>> > >> >> > >> > >> >>> > >> >> > >> The most important aspect of Kafka is compatibility, > and > >> the > >> >>> > >> >> > undocumented > >> >>> > >> >> > >> behavior has been in place for some time [0][1]. This > >> means > >> >>> > >> there’s no > >> >>> > >> >> > need > >> >>> > >> >> > >> to rush your improvement for 3.9, as we’ll need to > >> explicitly > >> >>> add > >> >>> > >> >> > default > >> >>> > >> >> > >> values after applying your patch to ensure we generate > >> the > >> >>> same > >> >>> > >> binary > >> >>> > >> >> > data. > >> >>> > >> >> > >> > >> >>> > >> >> > >> In short, we can improve the documentation first. In > the > >> >>> meantime, > >> >>> > >> we > >> >>> > >> >> > can > >> >>> > >> >> > >> continue discussing behavior clarification for 4.0, and > >> RM > >> >>> can keep > >> >>> > >> >> > running > >> >>> > >> >> > >> the RC for 3.9. Everything is on track. > >> >>> > >> >> > >> > >> >>> > >> >> > >> Best, > >> >>> > >> >> > >> Chia-Ping > >> >>> > >> >> > >> > >> >>> > >> >> > >> [0] > >> >>> > >> >> > >> > >> >>> > >> >> > > >> >>> > >> >> > >> >>> > >> > >> >>> > >> > https://github.com/apache/kafka/blob/3.8/clients/src/main/resources/common/message/FetchSnapshotResponse.json#L43 > >> >>> > >> >> > >> [1] > >> >>> > >> >> > >> > >> >>> > >> >> > > >> >>> > >> >> > >> >>> > >> > >> >>> > >> > https://github.com/apache/kafka/blob/3.8/group-coordinator/src/main/resources/common/message/ConsumerGroupMemberMetadataValue.json#L39 > >> >>> > >> >> > >> > >> >>> > >> >> > >> On 2024/10/27 15:28:05 Anton Agestam wrote: > >> >>> > >> >> > >> > -1, refer to comments on the RC 2 thread. > >> >>> > >> >> > >> > > >> >>> > >> >> > >> > Den sön 27 okt. 2024 kl 02:51 skrev Colin McCabe < > >> >>> > >> >> cmcc...@apache.org > >> >>> > >> >> > >: > >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > This is the RC5 candidate for the release of Apache > >> Kafka > >> >>> > >> 3.9.0. > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > - This is a major release, the final one in the 3.x > >> line. > >> >>> > >> (There > >> >>> > >> >> > may of > >> >>> > >> >> > >> > > course be other minor releases in this line, such > as > >> >>> 3.9.1.) > >> >>> > >> >> > >> > > - Tiered storage will be considered > production-ready > >> in > >> >>> this > >> >>> > >> >> > release. > >> >>> > >> >> > >> > > - This will be the final major release to feature > the > >> >>> > >> deprecated > >> >>> > >> >> > >> ZooKeeper > >> >>> > >> >> > >> > > mode. > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > This release includes the following KIPs: > >> >>> > >> >> > >> > > - KIP-853: Support dynamically changing KRaft > >> controller > >> >>> > >> >> membership > >> >>> > >> >> > >> > > - KIP-1057: Add remote log metadata flag to the > dump > >> log > >> >>> tool > >> >>> > >> >> > >> > > - KIP-1049: Add config log.summary.interval.ms to > >> Kafka > >> >>> > >> Streams > >> >>> > >> >> > >> > > - KIP-1040: Improve handling of nullable values in > >> >>> InsertField, > >> >>> > >> >> > >> > > ExtractField, and other transformations > >> >>> > >> >> > >> > > - KIP-1031: Control offset translation in > >> >>> MirrorSourceConnector > >> >>> > >> >> > >> > > - KIP-1033: Add Kafka Streams exception handler for > >> >>> exceptions > >> >>> > >> >> > >> occurring > >> >>> > >> >> > >> > > during processing > >> >>> > >> >> > >> > > - KIP-1017: Health check endpoint for Kafka Connect > >> >>> > >> >> > >> > > - KIP-1025: Optionally URL-encode clientID and > >> >>> clientSecret in > >> >>> > >> >> > >> > > authorization header > >> >>> > >> >> > >> > > - KIP-1005: Expose EarliestLocalOffset and > >> TieredOffset > >> >>> > >> >> > >> > > - KIP-950: Tiered Storage Disablement > >> >>> > >> >> > >> > > - KIP-956: Tiered Storage Quotas > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > Release notes for the 3.9.0 release: > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > >> >>> > >> >> > > >> >>> > >> >> > >> >>> > >> > >> >>> > >> > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/RELEASE_NOTES.html > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > *** Please download, test and vote by October 30, > >> 2024. > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > Kafka's KEYS file containing PGP keys we use to > sign > >> the > >> >>> > >> release: > >> >>> > >> >> > >> > > https://kafka.apache.org/KEYS > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > * Release artifacts to be voted upon (source and > >> binary): > >> >>> > >> >> > >> > > > >> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/ > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > * Docker release artifacts to be voted upon: > >> >>> > >> >> > >> > > apache/kafka:3.9.0-rc5 > >> >>> > >> >> > >> > > apache/kafka-native:3.9.0-rc5 > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > * Maven artifacts to be voted upon: > >> >>> > >> >> > >> > > > >> >>> > >> >> > > >> >>> > >> > >> >>> > >> https://repository.apache.org/content/groups/staging/org/apache/kafka/ > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > * Javadoc: > >> >>> > >> >> > >> > > > >> >>> > >> > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/javadoc/ > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > * Documentation: > >> >>> > >> >> > >> > > https://kafka.apache.org/39/documentation.html > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > * Protocol: > >> >>> > >> >> > >> > > https://kafka.apache.org/39/protocol.html > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > * Tag to be voted upon (off 3.9 branch) is the > >> 3.9.0-rc5 > >> >>> tag: > >> >>> > >> >> > >> > > > >> https://github.com/apache/kafka/releases/tag/3.9.0-rc5 > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > * Successful Docker Image Github Actions Pipeline > >> for 3.9 > >> >>> > >> branch: > >> >>> > >> >> > >> > > Docker Build Test Pipeline (JVM): > >> >>> > >> >> > >> > > > >> https://github.com/apache/kafka/actions/runs/11535300463 > >> >>> > >> >> > >> > > Docker Build Test Pipeline (Native): > >> >>> > >> >> > >> > > > >> https://github.com/apache/kafka/actions/runs/11535328957 > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > Thanks to everyone who helped with this release > >> candidate, > >> >>> > >> either > >> >>> > >> >> by > >> >>> > >> >> > >> > > contributing code, testing, or documentation. > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > Regards, > >> >>> > >> >> > >> > > Colin > >> >>> > >> >> > >> > > > >> >>> > >> >> > >> > > >> >>> > >> >> > >> > >> >>> > >> >> > > >> >>> > >> >> > >> >>> > >> > >> >>> > > > >> >>> > > > >> >>> > > -- > >> >>> > > David Arthur > >> >>> > >> > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
