It's difficult enough to balance kafka brokers with a single log
directory, not to mention attempting to juggle multiple ones. While JBOD
is great in terms of capacity, it's a pain in terms of management. After
6 months of constant manual reassignments I ended up going with RAID1+0
which is what LinkedIn uses as well as Confluent recommends.
Hats off to you if you manage to find a solution to this, just wanted to
share my painful experience.
On 2016.11.29 21:35, Tim Visher wrote:
Hello,
My kafka deploy has 5 servers with 3 log disks each. Over the weekend I
noticed that on 2 of the 5 servers the partitions appear to be imbalanced
amongst the log.dirs.
```
kafka3
/var/lib/kafka/disk1
3
/var/lib/kafka/disk2
3
/var/lib/kafka/disk3
3
kafka5
/var/lib/kafka/disk1
3
/var/lib/kafka/disk2
4
/var/lib/kafka/disk3
2
kafka1
/var/lib/kafka/disk1
3
/var/lib/kafka/disk2
3
/var/lib/kafka/disk3
3
kafka4
/var/lib/kafka/disk1
4
/var/lib/kafka/disk2
2
/var/lib/kafka/disk3
3
kafka2
/var/lib/kafka/disk1
3
/var/lib/kafka/disk2
3
/var/lib/kafka/disk3
3
```
You can see that 5 and 4 are both unbalanced.
Is there a reason for that? The partitions themselves are pretty much
perfectly balanced, but the directory chosen for them is not.
Is this an anti-pattern to be using multiple log.dirs per server?
Thanks in advance!
--
In Christ,
Timmy V.
http://blog.twonegatives.com/
http://five.sentenc.es/ -- Spend less time on mail
Best Regards
Karolis Pocius
IT System Engineer
Email: karolis.poc...@adform.com
Mobile: +370 620 22108
Sporto g. 18, LT-09238 Vilnius, Lithuania
Disclaimer:
The information contained in this message and attachments is intended
solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
