Hey Andrew,

Thanks for the reply. Currently, we are using the same method as you
described. Wanted to make sure if there is a better way.

It seems there isn't currently. So we will keep using this only.

On Tue, Oct 25, 2022 at 7:23 PM Andrew Grant <agr...@confluent.io.invalid>
wrote:

> Hey Lehar,
>
>
> I don’t think there’s a way to control this during topic creation. I just
> took a look through
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/AdminUtils.scala
> and it does appear partition assignment does not account for each broker’s
> different log directories. I also took a look at the kafka-topics.sh script
> and it has a --replica-assignment argument but that looks to only allow
> specifying brokers. During topic creation, once a replica has been chosen I
> think we then choose the directory with the fewest number of partitions -
> see
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogManager.scala#L1192
>
>
> What I think you can do is move existing partitions around with the
> kafka-reassign-partitions.sh script. From running the command locally:
>
>
> --reassignment-json-file <String:       The JSON file with the partition
>
>   manual assignment json file path>       reassignment configurationThe
> format
>
>                                           to use is -
>
>                                         {"partitions":
>
>                                         [{"topic": "foo",
>
>                                           "partition": 1,
>
>                                           "replicas": [1,2,3],
>
>                                           "log_dirs":
> ["dir1","dir2","dir3"]
>
>                                           }],
>
>                                         "version":1
>
>                                         }
>
>                                         Note that "log_dirs" is optional.
> When
>
>                                           it is specified, its length must
>
>                                           equal the length of the replicas
>
>                                           list. The value in this list can
> be
>
>                                           either "any" or the absolution
> path
>
>                                           of the log directory on the
> broker.
>
>                                           If absolute log directory path is
>
>                                           specified, the replica will be
> moved
>
>                                           to the specified log directory on
>
>                                           the broker.
>
>
> There’s the log_dirs field you can use in the JSON file to move partitions
> between directories.
>
>
> Hope that helps a bit.
>
>
> Andrew
>
> On Tue, Oct 25, 2022 at 6:56 AM Lehar Jain <leha...@media.net.invalid>
> wrote:
>
> > Hey,
> >
> > We run Kafka brokers with multiple log directories. I wanted to know how
> > Kafka balances traffic between various directories. Can we have our own
> > strategy to distribute different partitions to different directories. As
> > currently, we are facing an imbalance in sizes of the aforementioned
> > directories, some directories have a lot of empty space whereas others
> are
> > getting filled quickly.
> >
> >
> > Regards
> >
>

Reply via email to