Hi Jakub, It has always been required to separate control plane listeners and controller listeners. Failing to do this is a major misconfiguration. It may not have been caught sometimes, but that is a bug.
It should be simple to fix the configuration you posted -- simply have a different name for the controller listener than the control plane listener. best, Colin On Mon, Oct 14, 2024, at 11:16, Jakub Scholz wrote: > The different name of the controller listener for KRaft controllers and > control plane listener in ZooKeeper-based cluster was not required before > and it is not simple to change to handle now at the "last minute". So given > that this is called production-ready already for some time, I think this is > breaking API change and should be treated as such. > > Thanks & Regards > Jakub > > On Mon, Oct 14, 2024 at 7:55 PM Colin McCabe <cmcc...@apache.org> wrote: > >> Hi Jakub, >> >> After looking through the attached file on the JIRA, I can say that this >> is a misconfiguration. control.plane.listener is a totally separate concept >> from control.plane.listener.name. They should never be set to the same >> value. The controller listener must have a different name and value than >> the control plane listener (if any). >> >> I also tested myself that KRaft migration works with >> control.plane.listener configured. It works on both 3.8 and 3.9-RC2. >> >> My initial statement that control.plane.listener was not supported during >> ZK migration was incorrect. As you said, it is supported during migration >> up to the point that we are in KRaft mode. (Another reason why having the >> control plane listener = controller listener would not make sense.) >> >> Thanks for the bug report and discussion. I've closed this as invalid now >> that I have tested migration using control.plane.listener for myself and >> verified that it works. >> >> best, >> Colin >> >> On Mon, Oct 14, 2024, at 08:31, Jakub Scholz wrote: >> >> control.plane.listener is not (and never has been) supported in KRaft >> > mode. >> > >> > You mean control.plane.listener.name is not supported in KRaft I guess? >> > Well, this is not KRaft, this is migration, so it uses the settings that >> it >> > used before for the Zoo-based cluster and that includes using dedicated >> > control plane listener. I don't think I can "just remove it" because the >> > other nodes will use it during the rolling update. >> > >> > This also worked fine with 3.8 (and 3.7, etc.) -> so if it is not >> supported >> > now, it is a breaking API change I guess which should be a blocker. >> > >> > Thanks & Regards >> > Jakub >> > >> > On Mon, Oct 14, 2024 at 5:12 PM Colin McCabe <cmcc...@apache.org> wrote: >> > >> >> Hi Jakub, >> >> >> >> Thanks for testing. control.plane.listener is not (and never has been) >> >> supported in KRaft mode. You have to remove control.plane.listener >> >> configurations before migrating. I filed KAFKA-17790 to document this in >> >> the migration instructions. (This is not a blocker for the release, >> though.) >> >> >> >> best, >> >> Colin >> >> >> >> On Mon, Oct 14, 2024, at 02:52, Jakub Scholz wrote: >> >> > Hi Colin, >> >> > >> >> > Thanks for the RC. I did some testing of it and run into >> >> > https://issues.apache.org/jira/browse/KAFKA-17788 which seems to be a >> >> > regression in the migration to KRaft process. >> >> > >> >> > Can someone who understands this part of the codebase look into it >> >> please? >> >> > >> >> > Thanks & Regards >> >> > Jakub >> >> > >> >> > On Thu, Oct 10, 2024 at 11:16 PM Colin McCabe <cmcc...@apache.org> >> >> wrote: >> >> > >> >> >> This is the second candidate for the release of Apache Kafka 3.9.0. I >> >> have >> >> >> titled it rc2 since I had an rc1 which got very far, even to the >> point >> >> of >> >> >> pushing tags and docker images, before I spotted an issue. So rather >> >> than >> >> >> mutate the tags, I decided to skip over rc1. >> >> >> >> >> >> - This is a major release, the final one in the 3.x line. (There may >> of >> >> >> course be other minor releases in this line, such as 3.9.1.) >> >> >> - Tiered storage will be considered production-ready in this release. >> >> >> - This will be the final major release to feature the deprecated >> >> ZooKeeper >> >> >> mode. >> >> >> >> >> >> This release includes the following KIPs: >> >> >> - KIP-853: Support dynamically changing KRaft controller membership >> >> >> - KIP-1057: Add remote log metadata flag to the dump log tool >> >> >> - KIP-1049: Add config log.summary.interval.ms to Kafka Streams >> >> >> - KIP-1040: Improve handling of nullable values in InsertField, >> >> >> ExtractField, and other transformations >> >> >> - KIP-1031: Control offset translation in MirrorSourceConnector >> >> >> - KIP-1033: Add Kafka Streams exception handler for exceptions >> occurring >> >> >> during processing >> >> >> - KIP-1017: Health check endpoint for Kafka Connect >> >> >> - KIP-1025: Optionally URL-encode clientID and clientSecret in >> >> >> authorization header >> >> >> - KIP-1005: Expose EarliestLocalOffset and TieredOffset >> >> >> - KIP-950: Tiered Storage Disablement >> >> >> - KIP-956: Tiered Storage Quotas >> >> >> >> >> >> Release notes for the 3.9.0 release: >> >> >> >> >> >> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc2/RELEASE_NOTES.html >> >> >> >> >> >> *** Please download, test and vote by October 16, 2024. >> >> >> >> >> >> Kafka's KEYS file containing PGP keys we use to sign the release: >> >> >> https://kafka.apache.org/KEYS >> >> >> >> >> >> * Release artifacts to be voted upon (source and binary): >> >> >> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc2/ >> >> >> >> >> >> * Docker release artifacts to be voted upon: >> >> >> apache/kafka:3.9.0-rc2 >> >> >> apache/kafka-native:3.9.0-rc2 >> >> >> >> >> >> * Maven artifacts to be voted upon: >> >> >> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/ >> >> >> >> >> >> * Javadoc: >> >> >> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc2/javadoc/ >> >> >> >> >> >> * Documentation: >> >> >> https://kafka.apache.org/39/documentation.html >> >> >> >> >> >> * Protocol: >> >> >> https://kafka.apache.org/39/protocol.html >> >> >> >> >> >> * Tag to be voted upon (off 3.9 branch) is the 3.9.0-rc2 tag: >> >> >> https://github.com/apache/kafka/releases/tag/3.9.0-rc2 >> >> >> >> >> >> * Successful Docker Image Github Actions Pipeline for 3.9 branch: >> >> >> Docker Build Test Pipeline (JVM): >> >> >> https://github.com/apache/kafka/actions/runs/11281563007 >> >> >> Docker Build Test Pipeline (Native): >> >> >> https://github.com/apache/kafka/actions/runs/11281608809 >> >> >> >> >> >> Thanks to everyone who helped with this release candidate, either by >> >> >> contributing code, testing, or documentation. >> >> >> >> >> >> Regards, >> >> >> Colin >> >> >> >> >> >>
