So if I understand this discussion correctly, there is no clear path to scale down cluster is it?
On Tue, 29 Oct 2024 at 7:40 AM, Luke Chen <show...@gmail.com> wrote: > Hi hayoung, > > This is the problem that KIP-1066 > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1066%3A+Mechanism+to+cordon+brokers+and+log+directories > > > is trying to address. > It is under discussion, and should be included in the upcoming releases. > > Thanks. > Luke > > > > On Tue, Oct 29, 2024 at 12:45 AM hayoung lee <l4314...@gmail.com> wrote: > > > Hi Team, > > > > I am currently operating a Kafka cluster in KRaft mode and would like to > > raise a few questions and suggestions regarding the cluster scale-down > > process. > > > > Recently, I scaled down the cluster by removing one broker; however, I > > encountered a situation where the removed broker still appeared in the > > cluster, causing new partitions to be assigned to it. To address this, I > > was able to remove the broker from the controller metadata using the > > kafka-cluster.sh command. However, I could not find any documentation on > > this method, so I had to refer to the Kafka source code to discover this > > solution. The issue is now resolved, but I would like to ask if there is > a > > more streamlined approach for scaling down. > > > > Additional Questions and Suggestions: > > > > 1. > > > > *Automatic Cluster Exclusion* > > I initially expected that a removed broker would be automatically > > excluded from the cluster after a certain period. However, the broker > > remained part of the cluster for quite some time. I am curious whether > > Kafka has a feature that automatically detects and excludes removed > > brokers > > from the cluster. > > 2. > > > > *Atomic Scale Down* > > If the above feature is not available, I would like to inquire if > there > > is a way to atomically exclude a broker and simultaneously remove it > > from > > the cluster metadata. Since we manage a Kafka cluster with continuous > > topic > > creation, a non-atomic process might lead to new topics being assigned > > to > > the broker between its exclusion and metadata removal, necessitating > > manual > > partition reassignments. > > 3. > > > > *Documentation Enhancement (Suggestion)* > > While addressing the scale-down issue, I could not find any relevant > > information in the Kafka User Guide. I had to refer to the source > code, > > specifically the unregister functionality, to learn that this could be > > done using kafka-cluster.sh. Including documentation on this process > > would likely make it easier for other users to manage their clusters > > effectively. > > > > Thank you, and I look forward to any guidance or recommendations. > > > > Best regards, > > > > hayoung > > >
