So if I understand this discussion correctly, there is no clear path to
scale down cluster is it?



On Tue, 29 Oct 2024 at 7:40 AM, Luke Chen <show...@gmail.com> wrote:

> Hi hayoung,
>
> This is the problem that KIP-1066
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1066%3A+Mechanism+to+cordon+brokers+and+log+directories
> >
> is trying to address.
> It is under discussion, and should be included in the upcoming releases.
>
> Thanks.
> Luke
>
>
>
> On Tue, Oct 29, 2024 at 12:45 AM hayoung lee <l4314...@gmail.com> wrote:
>
> > Hi Team,
> >
> > I am currently operating a Kafka cluster in KRaft mode and would like to
> > raise a few questions and suggestions regarding the cluster scale-down
> > process.
> >
> > Recently, I scaled down the cluster by removing one broker; however, I
> > encountered a situation where the removed broker still appeared in the
> > cluster, causing new partitions to be assigned to it. To address this, I
> > was able to remove the broker from the controller metadata using the
> > kafka-cluster.sh command. However, I could not find any documentation on
> > this method, so I had to refer to the Kafka source code to discover this
> > solution. The issue is now resolved, but I would like to ask if there is
> a
> > more streamlined approach for scaling down.
> >
> > Additional Questions and Suggestions:
> >
> >    1.
> >
> >    *Automatic Cluster Exclusion*
> >    I initially expected that a removed broker would be automatically
> >    excluded from the cluster after a certain period. However, the broker
> >    remained part of the cluster for quite some time. I am curious whether
> >    Kafka has a feature that automatically detects and excludes removed
> > brokers
> >    from the cluster.
> >    2.
> >
> >    *Atomic Scale Down*
> >    If the above feature is not available, I would like to inquire if
> there
> >    is a way to atomically exclude a broker and simultaneously remove it
> > from
> >    the cluster metadata. Since we manage a Kafka cluster with continuous
> > topic
> >    creation, a non-atomic process might lead to new topics being assigned
> > to
> >    the broker between its exclusion and metadata removal, necessitating
> > manual
> >    partition reassignments.
> >    3.
> >
> >    *Documentation Enhancement (Suggestion)*
> >    While addressing the scale-down issue, I could not find any relevant
> >    information in the Kafka User Guide. I had to refer to the source
> code,
> >    specifically the unregister functionality, to learn that this could be
> >    done using kafka-cluster.sh. Including documentation on this process
> >    would likely make it easier for other users to manage their clusters
> >    effectively.
> >
> > Thank you, and I look forward to any guidance or recommendations.
> >
> > Best regards,
> >
> > hayoung
> >
>

Reply via email to