Hi hayoung,

This is the problem that KIP-1066
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-1066%3A+Mechanism+to+cordon+brokers+and+log+directories>
is trying to address.
It is under discussion, and should be included in the upcoming releases.

Thanks.
Luke



On Tue, Oct 29, 2024 at 12:45 AM hayoung lee <l4314...@gmail.com> wrote:

> Hi Team,
>
> I am currently operating a Kafka cluster in KRaft mode and would like to
> raise a few questions and suggestions regarding the cluster scale-down
> process.
>
> Recently, I scaled down the cluster by removing one broker; however, I
> encountered a situation where the removed broker still appeared in the
> cluster, causing new partitions to be assigned to it. To address this, I
> was able to remove the broker from the controller metadata using the
> kafka-cluster.sh command. However, I could not find any documentation on
> this method, so I had to refer to the Kafka source code to discover this
> solution. The issue is now resolved, but I would like to ask if there is a
> more streamlined approach for scaling down.
>
> Additional Questions and Suggestions:
>
>    1.
>
>    *Automatic Cluster Exclusion*
>    I initially expected that a removed broker would be automatically
>    excluded from the cluster after a certain period. However, the broker
>    remained part of the cluster for quite some time. I am curious whether
>    Kafka has a feature that automatically detects and excludes removed
> brokers
>    from the cluster.
>    2.
>
>    *Atomic Scale Down*
>    If the above feature is not available, I would like to inquire if there
>    is a way to atomically exclude a broker and simultaneously remove it
> from
>    the cluster metadata. Since we manage a Kafka cluster with continuous
> topic
>    creation, a non-atomic process might lead to new topics being assigned
> to
>    the broker between its exclusion and metadata removal, necessitating
> manual
>    partition reassignments.
>    3.
>
>    *Documentation Enhancement (Suggestion)*
>    While addressing the scale-down issue, I could not find any relevant
>    information in the Kafka User Guide. I had to refer to the source code,
>    specifically the unregister functionality, to learn that this could be
>    done using kafka-cluster.sh. Including documentation on this process
>    would likely make it easier for other users to manage their clusters
>    effectively.
>
> Thank you, and I look forward to any guidance or recommendations.
>
> Best regards,
>
> hayoung
>

Reply via email to