There's Cruise Control, https://github.com/linkedin/cruise-control, which
is open-source and could help with automated balancing.

On Thu, Oct 27, 2022 at 10:26 AM <ga...@hotmail.co.uk> wrote:

> Auto rebalancing is a very important feature to run Kafka in a production
> environment. Given the confluent already have this feature, are there any
> space that the open source version could have this feature as well?
> Or, is it the idea that opensource version shouldn't be used in a high
> load production environment?
>
> ________________________________
> 发件人: sunil chaudhari <sunilmchaudhar...@gmail.com>
> 发送时间: 2022年10月27日 3:11
> 收件人: users@kafka.apache.org <users@kafka.apache.org>
> 主题: Re: Balancing traffic between multiple directories
>
> Hi Lehar,
> You are right. There is no better way in open source Kafka.
> However confluent has something called as Auto Rebalancing feature.
> Can you check if there is free version with this feature?
>
> It start balancing of  brokers automatically when it see there is uneven
> distribution of partitions.
>
> Regards,
> Sunil.
> On Wed, 26 Oct 2022 at 12:03 PM, Lehar Jain <leha...@media.net.invalid>
> wrote:
>
> > Hey Andrew,
> >
> > Thanks for the reply. Currently, we are using the same method as you
> > described. Wanted to make sure if there is a better way.
> >
> > It seems there isn't currently. So we will keep using this only.
> >
> > On Tue, Oct 25, 2022 at 7:23 PM Andrew Grant <agr...@confluent.io.invalid
> >
> > wrote:
> >
> > > Hey Lehar,
> > >
> > >
> > > I don’t think there’s a way to control this during topic creation. I
> just
> > > took a look through
> > >
> > >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/AdminUtils.scala
> > > and it does appear partition assignment does not account for each
> > broker’s
> > > different log directories. I also took a look at the kafka-topics.sh
> > script
> > > and it has a --replica-assignment argument but that looks to only allow
> > > specifying brokers. During topic creation, once a replica has been
> > chosen I
> > > think we then choose the directory with the fewest number of
> partitions -
> > > see
> > >
> > >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogManager.scala#L1192
> > >
> > >
> > > What I think you can do is move existing partitions around with the
> > > kafka-reassign-partitions.sh script. From running the command locally:
> > >
> > >
> > > --reassignment-json-file <String:       The JSON file with the
> partition
> > >
> > >   manual assignment json file path>       reassignment configurationThe
> > > format
> > >
> > >                                           to use is -
> > >
> > >                                         {"partitions":
> > >
> > >                                         [{"topic": "foo",
> > >
> > >                                           "partition": 1,
> > >
> > >                                           "replicas": [1,2,3],
> > >
> > >                                           "log_dirs":
> > > ["dir1","dir2","dir3"]
> > >
> > >                                           }],
> > >
> > >                                         "version":1
> > >
> > >                                         }
> > >
> > >                                         Note that "log_dirs" is
> optional.
> > > When
> > >
> > >                                           it is specified, its length
> > must
> > >
> > >                                           equal the length of the
> > replicas
> > >
> > >                                           list. The value in this list
> > can
> > > be
> > >
> > >                                           either "any" or the
> absolution
> > > path
> > >
> > >                                           of the log directory on the
> > > broker.
> > >
> > >                                           If absolute log directory
> path
> > is
> > >
> > >                                           specified, the replica will
> be
> > > moved
> > >
> > >                                           to the specified log
> directory
> > on
> > >
> > >                                           the broker.
> > >
> > >
> > > There’s the log_dirs field you can use in the JSON file to move
> > partitions
> > > between directories.
> > >
> > >
> > > Hope that helps a bit.
> > >
> > >
> > > Andrew
> > >
> > > On Tue, Oct 25, 2022 at 6:56 AM Lehar Jain <leha...@media.net.invalid>
> > > wrote:
> > >
> > > > Hey,
> > > >
> > > > We run Kafka brokers with multiple log directories. I wanted to know
> > how
> > > > Kafka balances traffic between various directories. Can we have our
> own
> > > > strategy to distribute different partitions to different directories.
> > As
> > > > currently, we are facing an imbalance in sizes of the aforementioned
> > > > directories, some directories have a lot of empty space whereas
> others
> > > are
> > > > getting filled quickly.
> > > >
> > > >
> > > > Regards
> > > >
> > >
> >
>

Reply via email to