Yes It seems that there is still no way to prevent partition assignment that may occur between broker shutdown and unregister (scale down). This topic seems to be under discussion in KIP-1066
On 2024/10/29 02:18:57 sunil chaudhari wrote: > So if I understand this discussion correctly, there is no clear path to > scale down cluster is it? > > > > On Tue, 29 Oct 2024 at 7:40 AM, Luke Chen <sh...@gmail.com> wrote: > > > Hi hayoung, > > > > This is the problem that KIP-1066 > > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1066%3A+Mechanism+to+cordon+brokers+and+log+directories > > > > > is trying to address. > > It is under discussion, and should be included in the upcoming releases. > > > > Thanks. > > Luke > > > > > > > > On Tue, Oct 29, 2024 at 12:45 AM hayoung lee <l4...@gmail.com> wrote: > > > > > Hi Team, > > > > > > I am currently operating a Kafka cluster in KRaft mode and would like to > > > raise a few questions and suggestions regarding the cluster scale-down > > > process. > > > > > > Recently, I scaled down the cluster by removing one broker; however, I > > > encountered a situation where the removed broker still appeared in the > > > cluster, causing new partitions to be assigned to it. To address this, I > > > was able to remove the broker from the controller metadata using the > > > kafka-cluster.sh command. However, I could not find any documentation on > > > this method, so I had to refer to the Kafka source code to discover this > > > solution. The issue is now resolved, but I would like to ask if there is > > a > > > more streamlined approach for scaling down. > > > > > > Additional Questions and Suggestions: > > > > > > 1. > > > > > > *Automatic Cluster Exclusion* > > > I initially expected that a removed broker would be automatically > > > excluded from the cluster after a certain period. However, the broker > > > remained part of the cluster for quite some time. I am curious whether > > > Kafka has a feature that automatically detects and excludes removed > > > brokers > > > from the cluster. > > > 2. > > > > > > *Atomic Scale Down* > > > If the above feature is not available, I would like to inquire if > > there > > > is a way to atomically exclude a broker and simultaneously remove it > > > from > > > the cluster metadata. Since we manage a Kafka cluster with continuous > > > topic > > > creation, a non-atomic process might lead to new topics being assigned > > > to > > > the broker between its exclusion and metadata removal, necessitating > > > manual > > > partition reassignments. > > > 3. > > > > > > *Documentation Enhancement (Suggestion)* > > > While addressing the scale-down issue, I could not find any relevant > > > information in the Kafka User Guide. I had to refer to the source > > code, > > > specifically the unregister functionality, to learn that this could be > > > done using kafka-cluster.sh. Including documentation on this process > > > would likely make it easier for other users to manage their clusters > > > effectively. > > > > > > Thank you, and I look forward to any guidance or recommendations. > > > > > > Best regards, > > > > > > hayoung > > > > > >
