Hi Colin, So, how exactly does this major misconfiguration that was not documented for over a year and nobody complained manifest itself? What should I look for in the logs? What are the problems it manifests itself through? There are plenty of users who went through the migration with this "major misconfiguration". So what should they be looking for? Did they lose messages? Does it have security consequences that require a CVE? How does one recover from the problems it caused?
I do not think changing the controller name is as simple as you suggest. And it is especially not simple if you need to roll it out to thousands of users. I also feel that issues like this need to be taken more seriously than just shrugging it off and quickly closing JIRA someone else opened as invalid. This is not the first thing that was found as undocumented. It indicates significant quality issues, at least on the documentation side. How many other "major misconfigurations" are still left? Jakub On Mon, Oct 14, 2024 at 11:33 PM Colin McCabe <cmcc...@apache.org> wrote: > Hi Jakub, > > It has always been required to separate control plane listeners and > controller listeners. Failing to do this is a major misconfiguration. It > may not have been caught sometimes, but that is a bug. > > It should be simple to fix the configuration you posted -- simply have a > different name for the controller listener than the control plane listener. > > best, > Colin > > On Mon, Oct 14, 2024, at 11:16, Jakub Scholz wrote: > > The different name of the controller listener for KRaft controllers and > > control plane listener in ZooKeeper-based cluster was not required before > > and it is not simple to change to handle now at the "last minute". So > given > > that this is called production-ready already for some time, I think this > is > > breaking API change and should be treated as such. > > > > Thanks & Regards > > Jakub > > > > On Mon, Oct 14, 2024 at 7:55 PM Colin McCabe <cmcc...@apache.org> wrote: > > > >> Hi Jakub, > >> > >> After looking through the attached file on the JIRA, I can say that this > >> is a misconfiguration. control.plane.listener is a totally separate > concept > >> from control.plane.listener.name. They should never be set to the same > >> value. The controller listener must have a different name and value than > >> the control plane listener (if any). > >> > >> I also tested myself that KRaft migration works with > >> control.plane.listener configured. It works on both 3.8 and 3.9-RC2. > >> > >> My initial statement that control.plane.listener was not supported > during > >> ZK migration was incorrect. As you said, it is supported during > migration > >> up to the point that we are in KRaft mode. (Another reason why having > the > >> control plane listener = controller listener would not make sense.) > >> > >> Thanks for the bug report and discussion. I've closed this as invalid > now > >> that I have tested migration using control.plane.listener for myself and > >> verified that it works. > >> > >> best, > >> Colin > >> > >> On Mon, Oct 14, 2024, at 08:31, Jakub Scholz wrote: > >> >> control.plane.listener is not (and never has been) supported in KRaft > >> > mode. > >> > > >> > You mean control.plane.listener.name is not supported in KRaft I > guess? > >> > Well, this is not KRaft, this is migration, so it uses the settings > that > >> it > >> > used before for the Zoo-based cluster and that includes using > dedicated > >> > control plane listener. I don't think I can "just remove it" because > the > >> > other nodes will use it during the rolling update. > >> > > >> > This also worked fine with 3.8 (and 3.7, etc.) -> so if it is not > >> supported > >> > now, it is a breaking API change I guess which should be a blocker. > >> > > >> > Thanks & Regards > >> > Jakub > >> > > >> > On Mon, Oct 14, 2024 at 5:12 PM Colin McCabe <cmcc...@apache.org> > wrote: > >> > > >> >> Hi Jakub, > >> >> > >> >> Thanks for testing. control.plane.listener is not (and never has > been) > >> >> supported in KRaft mode. You have to remove control.plane.listener > >> >> configurations before migrating. I filed KAFKA-17790 to document > this in > >> >> the migration instructions. (This is not a blocker for the release, > >> though.) > >> >> > >> >> best, > >> >> Colin > >> >> > >> >> On Mon, Oct 14, 2024, at 02:52, Jakub Scholz wrote: > >> >> > Hi Colin, > >> >> > > >> >> > Thanks for the RC. I did some testing of it and run into > >> >> > https://issues.apache.org/jira/browse/KAFKA-17788 which seems to > be a > >> >> > regression in the migration to KRaft process. > >> >> > > >> >> > Can someone who understands this part of the codebase look into it > >> >> please? > >> >> > > >> >> > Thanks & Regards > >> >> > Jakub > >> >> > > >> >> > On Thu, Oct 10, 2024 at 11:16 PM Colin McCabe <cmcc...@apache.org> > >> >> wrote: > >> >> > > >> >> >> This is the second candidate for the release of Apache Kafka > 3.9.0. I > >> >> have > >> >> >> titled it rc2 since I had an rc1 which got very far, even to the > >> point > >> >> of > >> >> >> pushing tags and docker images, before I spotted an issue. So > rather > >> >> than > >> >> >> mutate the tags, I decided to skip over rc1. > >> >> >> > >> >> >> - This is a major release, the final one in the 3.x line. (There > may > >> of > >> >> >> course be other minor releases in this line, such as 3.9.1.) > >> >> >> - Tiered storage will be considered production-ready in this > release. > >> >> >> - This will be the final major release to feature the deprecated > >> >> ZooKeeper > >> >> >> mode. > >> >> >> > >> >> >> This release includes the following KIPs: > >> >> >> - KIP-853: Support dynamically changing KRaft controller > membership > >> >> >> - KIP-1057: Add remote log metadata flag to the dump log tool > >> >> >> - KIP-1049: Add config log.summary.interval.ms to Kafka Streams > >> >> >> - KIP-1040: Improve handling of nullable values in InsertField, > >> >> >> ExtractField, and other transformations > >> >> >> - KIP-1031: Control offset translation in MirrorSourceConnector > >> >> >> - KIP-1033: Add Kafka Streams exception handler for exceptions > >> occurring > >> >> >> during processing > >> >> >> - KIP-1017: Health check endpoint for Kafka Connect > >> >> >> - KIP-1025: Optionally URL-encode clientID and clientSecret in > >> >> >> authorization header > >> >> >> - KIP-1005: Expose EarliestLocalOffset and TieredOffset > >> >> >> - KIP-950: Tiered Storage Disablement > >> >> >> - KIP-956: Tiered Storage Quotas > >> >> >> > >> >> >> Release notes for the 3.9.0 release: > >> >> >> > >> >> > >> > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc2/RELEASE_NOTES.html > >> >> >> > >> >> >> *** Please download, test and vote by October 16, 2024. > >> >> >> > >> >> >> Kafka's KEYS file containing PGP keys we use to sign the release: > >> >> >> https://kafka.apache.org/KEYS > >> >> >> > >> >> >> * Release artifacts to be voted upon (source and binary): > >> >> >> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc2/ > >> >> >> > >> >> >> * Docker release artifacts to be voted upon: > >> >> >> apache/kafka:3.9.0-rc2 > >> >> >> apache/kafka-native:3.9.0-rc2 > >> >> >> > >> >> >> * Maven artifacts to be voted upon: > >> >> >> > >> https://repository.apache.org/content/groups/staging/org/apache/kafka/ > >> >> >> > >> >> >> * Javadoc: > >> >> >> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc2/javadoc/ > >> >> >> > >> >> >> * Documentation: > >> >> >> https://kafka.apache.org/39/documentation.html > >> >> >> > >> >> >> * Protocol: > >> >> >> https://kafka.apache.org/39/protocol.html > >> >> >> > >> >> >> * Tag to be voted upon (off 3.9 branch) is the 3.9.0-rc2 tag: > >> >> >> https://github.com/apache/kafka/releases/tag/3.9.0-rc2 > >> >> >> > >> >> >> * Successful Docker Image Github Actions Pipeline for 3.9 branch: > >> >> >> Docker Build Test Pipeline (JVM): > >> >> >> https://github.com/apache/kafka/actions/runs/11281563007 > >> >> >> Docker Build Test Pipeline (Native): > >> >> >> https://github.com/apache/kafka/actions/runs/11281608809 > >> >> >> > >> >> >> Thanks to everyone who helped with this release candidate, either > by > >> >> >> contributing code, testing, or documentation. > >> >> >> > >> >> >> Regards, > >> >> >> Colin > >> >> >> > >> >> > >> >
