Hey All, It seems like I ran into an issue. While trying to do the migration (3.9 ZK -> 3.9 KRaft), at the very last step the whole cluster (except the controller I just restarted) fails with a fatal error where a controller (I think) requests a truncation that is below the HWM. I tried to capture everything but with the configs and the steps I described it can be reproduced every time.
I started looking into it, but since I'm completely unfamiliar with many parts of KRaft, I'd be happy if someone else more familiar with it could look at it. Especially if I didn't follow the migration procedure well (but I think I did so). All the info captured here: https://issues.apache.org/jira/browse/KAFKA-17950 Viktor On Fri, Nov 1, 2024 at 11:00 PM Colin McCabe <cmcc...@apache.org> wrote: > Argh. That should read "5 +1 votes". :) > > Thanks, everyone. > Colin > > > On Fri, Nov 1, 2024, at 14:58, Colin McCabe wrote: > > Hi all, > > > > This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes. > > > > +1 votes > > PMC Members: > > * Justine Olshan > > * Bill Bejeck > > * David Arthur > > * Luke Chen > > > > Community: > > * Federico Valeri > > > > 0 votes > > * No votes > > > > -1 votes > > * No votes > > > > Vote thread: > > https://lists.apache.org/thread/og8l24z8oj87x0zsoqddfxo54qsj4d1o > > > > I'll continue with the release process and the release announcement > > will follow in the next few days. > > > > best, > > Colin > > > > > > On Thu, Oct 31, 2024, at 21:03, Luke Chen wrote: > >> Hi Colin, > >> > >> Thanks for the update. > >> I've ran a sanity check as Mickael suggested for the maven artifacts > with > >> the one in https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/. > >> I used jarcomp < > https://activityworkshop.net/software/jarcomp/index.html> > >> to verify: > >> 1. kafka_2.13-3.9.0.jar > >> 2. kafka-clients-3.9.0.jar > >> 3. kafka-metadata-3.9.0.jar > >> > >> I confirmed all the class files in the jars have the same content. > >> > >> So +1 (binding) from me. > >> > >> Thank you. > >> Luke > >> > >> On Fri, Nov 1, 2024 at 1:08 AM Mickael Maison <mickael.mai...@gmail.com > > > >> wrote: > >> > >>> Hi, > >>> > >>> Can we add sanity checks to ensure all artifacts are present when > >>> running the release? > >>> Also should we use a different folder for each RC? This would > >>> eliminate the risk of artifacts from previous RCs hiding an issue. > >>> Luke spotted the old artifacts this time but this could have been > >>> missed. > >>> > >>> Thanks, > >>> Mickael > >>> > >>> On Thu, Oct 31, 2024 at 5:55 PM Colin McCabe <cmcc...@apache.org> > wrote: > >>> > > >>> > Hi David, > >>> > > >>> > Thanks for checking. I took a look and yes, I did forget to upload > the > >>> Scala 2.12 artifacts previously. The script runs gradlewAll, but when > I was > >>> trying to do the steps manually, I missed that and just ran gradlew, > which > >>> only did Scala 2.13. > >>> > > >>> > This should be fixed now! > >>> > > >>> > The scala 2.12 artifacts are here: > >>> > > >>> > > https://repository.apache.org/content/repositories/orgapachekafka-1827/ > >>> > > >>> > and also here: > >>> > > >>> > > >>> > https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.12/3.9.0/ > >>> > > >>> > I wonder if we should change the release.py script to just run the > >>> gradlew steps manually. The gradlewAll script seems a bit confusing (or > >>> maybe that's just me?) > >>> > > >>> > best, > >>> > Colin > >>> > > >>> > > >>> > On Wed, Oct 30, 2024, at 11:02, David Arthur wrote: > >>> > > Should there be Scala 2.12 artifacts in the Maven staging? > >>> > > > >>> > > I see these for 2.13: > >>> > > * > >>> > > > >>> > https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.13/3.9.0/ > >>> > > * > >>> > > > >>> > https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka-streams-scala_2.13/3.9.0/ > >>> > > > >>> > > but I don't see 2.12 for 3.9.0 release. > >>> > > > >>> > > I can confirm that the other artifacts (such as kafka-clients) do > have > >>> > > today's timestamp for the 3.9.0 artifact. > >>> > > > >>> > > -David > >>> > > > >>> > > On Wed, Oct 30, 2024 at 12:36 PM Colin McCabe <cmcc...@apache.org> > >>> wrote: > >>> > > > >>> > >> Thanks for checking, Luke. I did actually visit > >>> > >> https://repository.apache.org/#stagingRepositories and hit close > on > >>> the > >>> > >> artifacts as the release.py script says to do. I don't know why > they > >>> ended > >>> > >> up not showing up. > >>> > >> > >>> > >> Today I dropped all the maven artifacts for previous 3.9 RCs, > >>> switched to > >>> > >> java 8, and re-ran "gradlew publish" and "mvn deploy > -Pgpg-signing". > >>> Then I > >>> > >> closed the two artifacts that were uploaded. You can see them > here: > >>> > >> > >>> > >> > >>> > https://repository.apache.org/content/repositories/orgapachekafka-1825/ > >>> > >> > >>> > https://repository.apache.org/content/repositories/orgapachekafka-1826/ > >>> > >> > >>> > >> Hopefully these should now show up in Maven as expected. > >>> > >> > >>> > >> best, > >>> > >> Colin > >>> > >> > >>> > >> > >>> > >> On Tue, Oct 29, 2024, at 23:51, Luke Chen wrote: > >>> > >> > Hi Colin, > >>> > >> > > >>> > >> > I was trying to test the RC, but I found the artifacts in Maven > >>> artifacts > >>> > >> > to be voted upon are not up-to-date. > >>> > >> > Not only the "Last Modified" time in the 3.9.0 artifacts are on > >>> Oct. 10, > >>> > >> > but also the source code in the artifacts didn't include the > latest > >>> > >> commits > >>> > >> > in RC5 here <https://github.com/apache/kafka/commits/3.9.0-rc5/ > >. > >>> > >> > > >>> > >> > Could you help verify it? > >>> > >> > > >>> > >> > But I confirmed the release artifacts in " > >>> > >> > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/"; is > >>> up-to-date. > >>> > >> > I did: > >>> > >> > - checked the checksums and signatures > >>> > >> > - ran upgrade from v3.8.1 to v3.9.0 > >>> > >> > - ran quickstart with the 2.13 binaries > >>> > >> > - verify tiered storage and KIP-950 (dynamically disablement) > >>> > >> > > >>> > >> > I'll be +1 after the maven artifacts are updated. > >>> > >> > > >>> > >> > Thank you for running the release. > >>> > >> > Luke > >>> > >> > > >>> > >> > > >>> > >> > On Wed, Oct 30, 2024 at 5:40 AM Justine Olshan > >>> > >> <jols...@confluent.io.invalid> > >>> > >> > wrote: > >>> > >> > > >>> > >> >> Hey Colin, > >>> > >> >> > >>> > >> >> Thanks for the new RC. I've tested the --feature functionality > >>> that was > >>> > >> >> missing before. Seems to be working as expected now. > >>> > >> >> As a side note, I was looking into how to configure kip-853 > and it > >>> took > >>> > >> me > >>> > >> >> a moment to find the instructions in the documentation, I > wonder > >>> if it > >>> > >> >> could be included in the notable changes section of the upgrade > >>> notes. > >>> > >> Not > >>> > >> >> a blocker as I believe we can still update documentation. > >>> > >> >> > >>> > >> >> I did some other spot checks on the rest of the release. +1 > >>> (binding) > >>> > >> from > >>> > >> >> me > >>> > >> >> > >>> > >> >> Justine > >>> > >> >> > >>> > >> >> On Tue, Oct 29, 2024 at 12:45 PM Colin McCabe < > cmcc...@apache.org> > >>> > >> wrote: > >>> > >> >> > >>> > >> >> > Thanks, Anton. And thanks Chia-Ping Tsai for taking a look at > >>> how we > >>> > >> can > >>> > >> >> > improve the docs here... > >>> > >> >> > > >>> > >> >> > best, > >>> > >> >> > Colin > >>> > >> >> > > >>> > >> >> > > >>> > >> >> > On Tue, Oct 29, 2024, at 02:39, Anton Agestam wrote: > >>> > >> >> > > Hi Chia-Ping, > >>> > >> >> > > > >>> > >> >> > > Thanks for pointing those two fields out. I retract my -1. > >>> > >> >> > > > >>> > >> >> > > Cheers, > >>> > >> >> > > Anton > >>> > >> >> > > > >>> > >> >> > > Den sön 27 okt. 2024 kl 17:40 skrev Chia-Ping Tsai < > >>> > >> >> chia7...@apache.org > >>> > >> >> > >: > >>> > >> >> > > > >>> > >> >> > >> hi Anton > >>> > >> >> > >> > >>> > >> >> > >> Thanks for sharing your insights on Kafka > serialization—it’s > >>> really > >>> > >> >> cool > >>> > >> >> > >> and interesting to me. Additionally, you inspired me to > file > >>> a JIRA > >>> > >> >> > issue > >>> > >> >> > >> (KAFKA-17882) to improve the documentation. > >>> > >> >> > >> > >>> > >> >> > >> The most important aspect of Kafka is compatibility, and > the > >>> > >> >> > undocumented > >>> > >> >> > >> behavior has been in place for some time [0][1]. This > means > >>> > >> there’s no > >>> > >> >> > need > >>> > >> >> > >> to rush your improvement for 3.9, as we’ll need to > explicitly > >>> add > >>> > >> >> > default > >>> > >> >> > >> values after applying your patch to ensure we generate the > >>> same > >>> > >> binary > >>> > >> >> > data. > >>> > >> >> > >> > >>> > >> >> > >> In short, we can improve the documentation first. In the > >>> meantime, > >>> > >> we > >>> > >> >> > can > >>> > >> >> > >> continue discussing behavior clarification for 4.0, and RM > >>> can keep > >>> > >> >> > running > >>> > >> >> > >> the RC for 3.9. Everything is on track. > >>> > >> >> > >> > >>> > >> >> > >> Best, > >>> > >> >> > >> Chia-Ping > >>> > >> >> > >> > >>> > >> >> > >> [0] > >>> > >> >> > >> > >>> > >> >> > > >>> > >> >> > >>> > >> > >>> > https://github.com/apache/kafka/blob/3.8/clients/src/main/resources/common/message/FetchSnapshotResponse.json#L43 > >>> > >> >> > >> [1] > >>> > >> >> > >> > >>> > >> >> > > >>> > >> >> > >>> > >> > >>> > https://github.com/apache/kafka/blob/3.8/group-coordinator/src/main/resources/common/message/ConsumerGroupMemberMetadataValue.json#L39 > >>> > >> >> > >> > >>> > >> >> > >> On 2024/10/27 15:28:05 Anton Agestam wrote: > >>> > >> >> > >> > -1, refer to comments on the RC 2 thread. > >>> > >> >> > >> > > >>> > >> >> > >> > Den sön 27 okt. 2024 kl 02:51 skrev Colin McCabe < > >>> > >> >> cmcc...@apache.org > >>> > >> >> > >: > >>> > >> >> > >> > > >>> > >> >> > >> > > This is the RC5 candidate for the release of Apache > Kafka > >>> > >> 3.9.0. > >>> > >> >> > >> > > > >>> > >> >> > >> > > - This is a major release, the final one in the 3.x > line. > >>> > >> (There > >>> > >> >> > may of > >>> > >> >> > >> > > course be other minor releases in this line, such as > >>> 3.9.1.) > >>> > >> >> > >> > > - Tiered storage will be considered production-ready > in > >>> this > >>> > >> >> > release. > >>> > >> >> > >> > > - This will be the final major release to feature the > >>> > >> deprecated > >>> > >> >> > >> ZooKeeper > >>> > >> >> > >> > > mode. > >>> > >> >> > >> > > > >>> > >> >> > >> > > This release includes the following KIPs: > >>> > >> >> > >> > > - KIP-853: Support dynamically changing KRaft > controller > >>> > >> >> membership > >>> > >> >> > >> > > - KIP-1057: Add remote log metadata flag to the dump > log > >>> tool > >>> > >> >> > >> > > - KIP-1049: Add config log.summary.interval.ms to > Kafka > >>> > >> Streams > >>> > >> >> > >> > > - KIP-1040: Improve handling of nullable values in > >>> InsertField, > >>> > >> >> > >> > > ExtractField, and other transformations > >>> > >> >> > >> > > - KIP-1031: Control offset translation in > >>> MirrorSourceConnector > >>> > >> >> > >> > > - KIP-1033: Add Kafka Streams exception handler for > >>> exceptions > >>> > >> >> > >> occurring > >>> > >> >> > >> > > during processing > >>> > >> >> > >> > > - KIP-1017: Health check endpoint for Kafka Connect > >>> > >> >> > >> > > - KIP-1025: Optionally URL-encode clientID and > >>> clientSecret in > >>> > >> >> > >> > > authorization header > >>> > >> >> > >> > > - KIP-1005: Expose EarliestLocalOffset and > TieredOffset > >>> > >> >> > >> > > - KIP-950: Tiered Storage Disablement > >>> > >> >> > >> > > - KIP-956: Tiered Storage Quotas > >>> > >> >> > >> > > > >>> > >> >> > >> > > Release notes for the 3.9.0 release: > >>> > >> >> > >> > > > >>> > >> >> > >> > >>> > >> >> > > >>> > >> >> > >>> > >> > >>> > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/RELEASE_NOTES.html > >>> > >> >> > >> > > > >>> > >> >> > >> > > *** Please download, test and vote by October 30, > 2024. > >>> > >> >> > >> > > > >>> > >> >> > >> > > Kafka's KEYS file containing PGP keys we use to sign > the > >>> > >> release: > >>> > >> >> > >> > > https://kafka.apache.org/KEYS > >>> > >> >> > >> > > > >>> > >> >> > >> > > * Release artifacts to be voted upon (source and > binary): > >>> > >> >> > >> > > > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/ > >>> > >> >> > >> > > > >>> > >> >> > >> > > * Docker release artifacts to be voted upon: > >>> > >> >> > >> > > apache/kafka:3.9.0-rc5 > >>> > >> >> > >> > > apache/kafka-native:3.9.0-rc5 > >>> > >> >> > >> > > > >>> > >> >> > >> > > * Maven artifacts to be voted upon: > >>> > >> >> > >> > > > >>> > >> >> > > >>> > >> > >>> https://repository.apache.org/content/groups/staging/org/apache/kafka/ > >>> > >> >> > >> > > > >>> > >> >> > >> > > * Javadoc: > >>> > >> >> > >> > > > >>> > >> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/javadoc/ > >>> > >> >> > >> > > > >>> > >> >> > >> > > * Documentation: > >>> > >> >> > >> > > https://kafka.apache.org/39/documentation.html > >>> > >> >> > >> > > > >>> > >> >> > >> > > * Protocol: > >>> > >> >> > >> > > https://kafka.apache.org/39/protocol.html > >>> > >> >> > >> > > > >>> > >> >> > >> > > * Tag to be voted upon (off 3.9 branch) is the > 3.9.0-rc5 > >>> tag: > >>> > >> >> > >> > > > https://github.com/apache/kafka/releases/tag/3.9.0-rc5 > >>> > >> >> > >> > > > >>> > >> >> > >> > > * Successful Docker Image Github Actions Pipeline for > 3.9 > >>> > >> branch: > >>> > >> >> > >> > > Docker Build Test Pipeline (JVM): > >>> > >> >> > >> > > > https://github.com/apache/kafka/actions/runs/11535300463 > >>> > >> >> > >> > > Docker Build Test Pipeline (Native): > >>> > >> >> > >> > > > https://github.com/apache/kafka/actions/runs/11535328957 > >>> > >> >> > >> > > > >>> > >> >> > >> > > Thanks to everyone who helped with this release > candidate, > >>> > >> either > >>> > >> >> by > >>> > >> >> > >> > > contributing code, testing, or documentation. > >>> > >> >> > >> > > > >>> > >> >> > >> > > Regards, > >>> > >> >> > >> > > Colin > >>> > >> >> > >> > > > >>> > >> >> > >> > > >>> > >> >> > >> > >>> > >> >> > > >>> > >> >> > >>> > >> > >>> > > > >>> > > > >>> > > -- > >>> > > David Arthur > >>> >
