Hey Lehar,
I don’t think there’s a way to control this during topic creation. I just took a look through https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/AdminUtils.scala and it does appear partition assignment does not account for each broker’s different log directories. I also took a look at the kafka-topics.sh script and it has a --replica-assignment argument but that looks to only allow specifying brokers. During topic creation, once a replica has been chosen I think we then choose the directory with the fewest number of partitions - see https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogManager.scala#L1192 What I think you can do is move existing partitions around with the kafka-reassign-partitions.sh script. From running the command locally: --reassignment-json-file <String: The JSON file with the partition manual assignment json file path> reassignment configurationThe format to use is - {"partitions": [{"topic": "foo", "partition": 1, "replicas": [1,2,3], "log_dirs": ["dir1","dir2","dir3"] }], "version":1 } Note that "log_dirs" is optional. When it is specified, its length must equal the length of the replicas list. The value in this list can be either "any" or the absolution path of the log directory on the broker. If absolute log directory path is specified, the replica will be moved to the specified log directory on the broker. There’s the log_dirs field you can use in the JSON file to move partitions between directories. Hope that helps a bit. Andrew On Tue, Oct 25, 2022 at 6:56 AM Lehar Jain <leha...@media.net.invalid> wrote: > Hey, > > We run Kafka brokers with multiple log directories. I wanted to know how > Kafka balances traffic between various directories. Can we have our own > strategy to distribute different partitions to different directories. As > currently, we are facing an imbalance in sizes of the aforementioned > directories, some directories have a lot of empty space whereas others are > getting filled quickly. > > > Regards >
