Ok, sorry for the noise. It seems like my setup was incorrectly configured, one controller had an incorrect voters config and that caused the issue. Retested it and it works now. But given this, I might as well just give a +1. :)
On Wed, Nov 6, 2024 at 2:53 PM Viktor Somogyi-Vass < viktor.somo...@cloudera.com> wrote: > Hey All, > > It seems like I ran into an issue. While trying to do the migration (3.9 > ZK -> 3.9 KRaft), at the very last step the whole cluster (except the > controller I just restarted) fails with a fatal error where a controller (I > think) requests a truncation that is below the HWM. > I tried to capture everything but with the configs and the steps I > described it can be reproduced every time. > > I started looking into it, but since I'm completely unfamiliar with many > parts of KRaft, I'd be happy if someone else more familiar with it could > look at it. Especially if I didn't follow the migration procedure well (but > I think I did so). > All the info captured here: > https://issues.apache.org/jira/browse/KAFKA-17950 > > Viktor > > > On Fri, Nov 1, 2024 at 11:00 PM Colin McCabe <cmcc...@apache.org> wrote: > >> Argh. That should read "5 +1 votes". :) >> >> Thanks, everyone. >> Colin >> >> >> On Fri, Nov 1, 2024, at 14:58, Colin McCabe wrote: >> > Hi all, >> > >> > This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes. >> > >> > +1 votes >> > PMC Members: >> > * Justine Olshan >> > * Bill Bejeck >> > * David Arthur >> > * Luke Chen >> > >> > Community: >> > * Federico Valeri >> > >> > 0 votes >> > * No votes >> > >> > -1 votes >> > * No votes >> > >> > Vote thread: >> > https://lists.apache.org/thread/og8l24z8oj87x0zsoqddfxo54qsj4d1o >> > >> > I'll continue with the release process and the release announcement >> > will follow in the next few days. >> > >> > best, >> > Colin >> > >> > >> > On Thu, Oct 31, 2024, at 21:03, Luke Chen wrote: >> >> Hi Colin, >> >> >> >> Thanks for the update. >> >> I've ran a sanity check as Mickael suggested for the maven artifacts >> with >> >> the one in https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/. >> >> I used jarcomp < >> https://activityworkshop.net/software/jarcomp/index.html> >> >> to verify: >> >> 1. kafka_2.13-3.9.0.jar >> >> 2. kafka-clients-3.9.0.jar >> >> 3. kafka-metadata-3.9.0.jar >> >> >> >> I confirmed all the class files in the jars have the same content. >> >> >> >> So +1 (binding) from me. >> >> >> >> Thank you. >> >> Luke >> >> >> >> On Fri, Nov 1, 2024 at 1:08 AM Mickael Maison < >> mickael.mai...@gmail.com> >> >> wrote: >> >> >> >>> Hi, >> >>> >> >>> Can we add sanity checks to ensure all artifacts are present when >> >>> running the release? >> >>> Also should we use a different folder for each RC? This would >> >>> eliminate the risk of artifacts from previous RCs hiding an issue. >> >>> Luke spotted the old artifacts this time but this could have been >> >>> missed. >> >>> >> >>> Thanks, >> >>> Mickael >> >>> >> >>> On Thu, Oct 31, 2024 at 5:55 PM Colin McCabe <cmcc...@apache.org> >> wrote: >> >>> > >> >>> > Hi David, >> >>> > >> >>> > Thanks for checking. I took a look and yes, I did forget to upload >> the >> >>> Scala 2.12 artifacts previously. The script runs gradlewAll, but when >> I was >> >>> trying to do the steps manually, I missed that and just ran gradlew, >> which >> >>> only did Scala 2.13. >> >>> > >> >>> > This should be fixed now! >> >>> > >> >>> > The scala 2.12 artifacts are here: >> >>> > >> >>> > >> https://repository.apache.org/content/repositories/orgapachekafka-1827/ >> >>> > >> >>> > and also here: >> >>> > >> >>> > >> >>> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.12/3.9.0/ >> >>> > >> >>> > I wonder if we should change the release.py script to just run the >> >>> gradlew steps manually. The gradlewAll script seems a bit confusing >> (or >> >>> maybe that's just me?) >> >>> > >> >>> > best, >> >>> > Colin >> >>> > >> >>> > >> >>> > On Wed, Oct 30, 2024, at 11:02, David Arthur wrote: >> >>> > > Should there be Scala 2.12 artifacts in the Maven staging? >> >>> > > >> >>> > > I see these for 2.13: >> >>> > > * >> >>> > > >> >>> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.13/3.9.0/ >> >>> > > * >> >>> > > >> >>> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka-streams-scala_2.13/3.9.0/ >> >>> > > >> >>> > > but I don't see 2.12 for 3.9.0 release. >> >>> > > >> >>> > > I can confirm that the other artifacts (such as kafka-clients) do >> have >> >>> > > today's timestamp for the 3.9.0 artifact. >> >>> > > >> >>> > > -David >> >>> > > >> >>> > > On Wed, Oct 30, 2024 at 12:36 PM Colin McCabe <cmcc...@apache.org >> > >> >>> wrote: >> >>> > > >> >>> > >> Thanks for checking, Luke. I did actually visit >> >>> > >> https://repository.apache.org/#stagingRepositories and hit >> close on >> >>> the >> >>> > >> artifacts as the release.py script says to do. I don't know why >> they >> >>> ended >> >>> > >> up not showing up. >> >>> > >> >> >>> > >> Today I dropped all the maven artifacts for previous 3.9 RCs, >> >>> switched to >> >>> > >> java 8, and re-ran "gradlew publish" and "mvn deploy >> -Pgpg-signing". >> >>> Then I >> >>> > >> closed the two artifacts that were uploaded. You can see them >> here: >> >>> > >> >> >>> > >> >> >>> >> https://repository.apache.org/content/repositories/orgapachekafka-1825/ >> >>> > >> >> >>> >> https://repository.apache.org/content/repositories/orgapachekafka-1826/ >> >>> > >> >> >>> > >> Hopefully these should now show up in Maven as expected. >> >>> > >> >> >>> > >> best, >> >>> > >> Colin >> >>> > >> >> >>> > >> >> >>> > >> On Tue, Oct 29, 2024, at 23:51, Luke Chen wrote: >> >>> > >> > Hi Colin, >> >>> > >> > >> >>> > >> > I was trying to test the RC, but I found the artifacts in Maven >> >>> artifacts >> >>> > >> > to be voted upon are not up-to-date. >> >>> > >> > Not only the "Last Modified" time in the 3.9.0 artifacts are on >> >>> Oct. 10, >> >>> > >> > but also the source code in the artifacts didn't include the >> latest >> >>> > >> commits >> >>> > >> > in RC5 here < >> https://github.com/apache/kafka/commits/3.9.0-rc5/>. >> >>> > >> > >> >>> > >> > Could you help verify it? >> >>> > >> > >> >>> > >> > But I confirmed the release artifacts in " >> >>> > >> > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/"; is >> >>> up-to-date. >> >>> > >> > I did: >> >>> > >> > - checked the checksums and signatures >> >>> > >> > - ran upgrade from v3.8.1 to v3.9.0 >> >>> > >> > - ran quickstart with the 2.13 binaries >> >>> > >> > - verify tiered storage and KIP-950 (dynamically disablement) >> >>> > >> > >> >>> > >> > I'll be +1 after the maven artifacts are updated. >> >>> > >> > >> >>> > >> > Thank you for running the release. >> >>> > >> > Luke >> >>> > >> > >> >>> > >> > >> >>> > >> > On Wed, Oct 30, 2024 at 5:40 AM Justine Olshan >> >>> > >> <jols...@confluent.io.invalid> >> >>> > >> > wrote: >> >>> > >> > >> >>> > >> >> Hey Colin, >> >>> > >> >> >> >>> > >> >> Thanks for the new RC. I've tested the --feature functionality >> >>> that was >> >>> > >> >> missing before. Seems to be working as expected now. >> >>> > >> >> As a side note, I was looking into how to configure kip-853 >> and it >> >>> took >> >>> > >> me >> >>> > >> >> a moment to find the instructions in the documentation, I >> wonder >> >>> if it >> >>> > >> >> could be included in the notable changes section of the >> upgrade >> >>> notes. >> >>> > >> Not >> >>> > >> >> a blocker as I believe we can still update documentation. >> >>> > >> >> >> >>> > >> >> I did some other spot checks on the rest of the release. +1 >> >>> (binding) >> >>> > >> from >> >>> > >> >> me >> >>> > >> >> >> >>> > >> >> Justine >> >>> > >> >> >> >>> > >> >> On Tue, Oct 29, 2024 at 12:45 PM Colin McCabe < >> cmcc...@apache.org> >> >>> > >> wrote: >> >>> > >> >> >> >>> > >> >> > Thanks, Anton. And thanks Chia-Ping Tsai for taking a look >> at >> >>> how we >> >>> > >> can >> >>> > >> >> > improve the docs here... >> >>> > >> >> > >> >>> > >> >> > best, >> >>> > >> >> > Colin >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > On Tue, Oct 29, 2024, at 02:39, Anton Agestam wrote: >> >>> > >> >> > > Hi Chia-Ping, >> >>> > >> >> > > >> >>> > >> >> > > Thanks for pointing those two fields out. I retract my -1. >> >>> > >> >> > > >> >>> > >> >> > > Cheers, >> >>> > >> >> > > Anton >> >>> > >> >> > > >> >>> > >> >> > > Den sön 27 okt. 2024 kl 17:40 skrev Chia-Ping Tsai < >> >>> > >> >> chia7...@apache.org >> >>> > >> >> > >: >> >>> > >> >> > > >> >>> > >> >> > >> hi Anton >> >>> > >> >> > >> >> >>> > >> >> > >> Thanks for sharing your insights on Kafka >> serialization—it’s >> >>> really >> >>> > >> >> cool >> >>> > >> >> > >> and interesting to me. Additionally, you inspired me to >> file >> >>> a JIRA >> >>> > >> >> > issue >> >>> > >> >> > >> (KAFKA-17882) to improve the documentation. >> >>> > >> >> > >> >> >>> > >> >> > >> The most important aspect of Kafka is compatibility, and >> the >> >>> > >> >> > undocumented >> >>> > >> >> > >> behavior has been in place for some time [0][1]. This >> means >> >>> > >> there’s no >> >>> > >> >> > need >> >>> > >> >> > >> to rush your improvement for 3.9, as we’ll need to >> explicitly >> >>> add >> >>> > >> >> > default >> >>> > >> >> > >> values after applying your patch to ensure we generate >> the >> >>> same >> >>> > >> binary >> >>> > >> >> > data. >> >>> > >> >> > >> >> >>> > >> >> > >> In short, we can improve the documentation first. In the >> >>> meantime, >> >>> > >> we >> >>> > >> >> > can >> >>> > >> >> > >> continue discussing behavior clarification for 4.0, and >> RM >> >>> can keep >> >>> > >> >> > running >> >>> > >> >> > >> the RC for 3.9. Everything is on track. >> >>> > >> >> > >> >> >>> > >> >> > >> Best, >> >>> > >> >> > >> Chia-Ping >> >>> > >> >> > >> >> >>> > >> >> > >> [0] >> >>> > >> >> > >> >> >>> > >> >> > >> >>> > >> >> >> >>> > >> >> >>> >> https://github.com/apache/kafka/blob/3.8/clients/src/main/resources/common/message/FetchSnapshotResponse.json#L43 >> >>> > >> >> > >> [1] >> >>> > >> >> > >> >> >>> > >> >> > >> >>> > >> >> >> >>> > >> >> >>> >> https://github.com/apache/kafka/blob/3.8/group-coordinator/src/main/resources/common/message/ConsumerGroupMemberMetadataValue.json#L39 >> >>> > >> >> > >> >> >>> > >> >> > >> On 2024/10/27 15:28:05 Anton Agestam wrote: >> >>> > >> >> > >> > -1, refer to comments on the RC 2 thread. >> >>> > >> >> > >> > >> >>> > >> >> > >> > Den sön 27 okt. 2024 kl 02:51 skrev Colin McCabe < >> >>> > >> >> cmcc...@apache.org >> >>> > >> >> > >: >> >>> > >> >> > >> > >> >>> > >> >> > >> > > This is the RC5 candidate for the release of Apache >> Kafka >> >>> > >> 3.9.0. >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > - This is a major release, the final one in the 3.x >> line. >> >>> > >> (There >> >>> > >> >> > may of >> >>> > >> >> > >> > > course be other minor releases in this line, such as >> >>> 3.9.1.) >> >>> > >> >> > >> > > - Tiered storage will be considered production-ready >> in >> >>> this >> >>> > >> >> > release. >> >>> > >> >> > >> > > - This will be the final major release to feature the >> >>> > >> deprecated >> >>> > >> >> > >> ZooKeeper >> >>> > >> >> > >> > > mode. >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > This release includes the following KIPs: >> >>> > >> >> > >> > > - KIP-853: Support dynamically changing KRaft >> controller >> >>> > >> >> membership >> >>> > >> >> > >> > > - KIP-1057: Add remote log metadata flag to the dump >> log >> >>> tool >> >>> > >> >> > >> > > - KIP-1049: Add config log.summary.interval.ms to >> Kafka >> >>> > >> Streams >> >>> > >> >> > >> > > - KIP-1040: Improve handling of nullable values in >> >>> InsertField, >> >>> > >> >> > >> > > ExtractField, and other transformations >> >>> > >> >> > >> > > - KIP-1031: Control offset translation in >> >>> MirrorSourceConnector >> >>> > >> >> > >> > > - KIP-1033: Add Kafka Streams exception handler for >> >>> exceptions >> >>> > >> >> > >> occurring >> >>> > >> >> > >> > > during processing >> >>> > >> >> > >> > > - KIP-1017: Health check endpoint for Kafka Connect >> >>> > >> >> > >> > > - KIP-1025: Optionally URL-encode clientID and >> >>> clientSecret in >> >>> > >> >> > >> > > authorization header >> >>> > >> >> > >> > > - KIP-1005: Expose EarliestLocalOffset and >> TieredOffset >> >>> > >> >> > >> > > - KIP-950: Tiered Storage Disablement >> >>> > >> >> > >> > > - KIP-956: Tiered Storage Quotas >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > Release notes for the 3.9.0 release: >> >>> > >> >> > >> > > >> >>> > >> >> > >> >> >>> > >> >> > >> >>> > >> >> >> >>> > >> >> >>> >> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/RELEASE_NOTES.html >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > *** Please download, test and vote by October 30, >> 2024. >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > Kafka's KEYS file containing PGP keys we use to sign >> the >> >>> > >> release: >> >>> > >> >> > >> > > https://kafka.apache.org/KEYS >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > * Release artifacts to be voted upon (source and >> binary): >> >>> > >> >> > >> > > >> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/ >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > * Docker release artifacts to be voted upon: >> >>> > >> >> > >> > > apache/kafka:3.9.0-rc5 >> >>> > >> >> > >> > > apache/kafka-native:3.9.0-rc5 >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > * Maven artifacts to be voted upon: >> >>> > >> >> > >> > > >> >>> > >> >> > >> >>> > >> >> >>> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/ >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > * Javadoc: >> >>> > >> >> > >> > > >> >>> > >> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/javadoc/ >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > * Documentation: >> >>> > >> >> > >> > > https://kafka.apache.org/39/documentation.html >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > * Protocol: >> >>> > >> >> > >> > > https://kafka.apache.org/39/protocol.html >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > * Tag to be voted upon (off 3.9 branch) is the >> 3.9.0-rc5 >> >>> tag: >> >>> > >> >> > >> > > >> https://github.com/apache/kafka/releases/tag/3.9.0-rc5 >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > * Successful Docker Image Github Actions Pipeline >> for 3.9 >> >>> > >> branch: >> >>> > >> >> > >> > > Docker Build Test Pipeline (JVM): >> >>> > >> >> > >> > > >> https://github.com/apache/kafka/actions/runs/11535300463 >> >>> > >> >> > >> > > Docker Build Test Pipeline (Native): >> >>> > >> >> > >> > > >> https://github.com/apache/kafka/actions/runs/11535328957 >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > Thanks to everyone who helped with this release >> candidate, >> >>> > >> either >> >>> > >> >> by >> >>> > >> >> > >> > > contributing code, testing, or documentation. >> >>> > >> >> > >> > > >> >>> > >> >> > >> > > Regards, >> >>> > >> >> > >> > > Colin >> >>> > >> >> > >> > > >> >>> > >> >> > >> > >> >>> > >> >> > >> >> >>> > >> >> > >> >>> > >> >> >> >>> > >> >> >>> > > >> >>> > > >> >>> > > -- >> >>> > > David Arthur >> >>> >> >
