Hi, LogManager.nextLogDir() has the logic for choosing which directory to use. The documentation of the method says: /** * Choose the next directory in which to create a log. Currently this is done * by calculating the number of partitions in each directory and then choosing the * data directory with the fewest partitions. */
A possible explanation for what you are seeing is that some of the directories were added recently and simply did not fill up yet. Kafka doesn't attempt to rebalance logs across disks or directories (although this could be a nice feature request, at least for a utility). Gwen On Thu, Dec 25, 2014 at 1:30 AM, Yury Ruchin <yuri.ruc...@gmail.com> wrote: > I also noticed that some disks do not contain any partitions which is even > more strange. Why does broker not use them, while other disks are > overloaded? > > 2014-12-25 12:07 GMT+03:00 Yury Ruchin <yuri.ruc...@gmail.com>: > > > Hi, > > > > With Kafka 0.8.1.1, I'm continuously running into the issue with no disk > > space remaining. I'm observing that on the problematic brokers partitions > > are distributed across several identical disks very unevenly. For > example, > > provided the fact that partitions have similar size, I see 6 partitions > > being written to one disk, while the other disks get only 1-3 partitions > > each. > > > > What could cause that? Is that possible to somehow affect the partition > > distribution? > > > > Thanks, > > Yury > > >
