The intention is to allow the use of multiple disks without RAID or logical volume management. We have found that there are a lot of downsides to RAID--in particular a huge throughput hit. Since we already have a parallelism model due to partitioning and a fault tolerance model with replication RAID doesn't actually buy much. With this feature you can directly mount multiple disks as their own directory and the server will randomly assign partitions to them.
Obviously this will only work well if there are enough high-throughput partitions to make load balance evenly (e.g. if you have only one big partition per server then this isn't going to work). -Jay On Wed, Jun 19, 2013 at 11:01 PM, Jason Rosenberg <j...@squareup.com> wrote: > is it possible for a partition to have multiple replicas on different > directories on the same broker? (hopefully no!) > > > On Wed, Jun 19, 2013 at 10:47 PM, Jun Rao <jun...@gmail.com> wrote: > >> It takes a comma separated list and partition replicas are randomly >> distributed to the list. >> >> Thanks, >> >> Jun >> >> >> On Wed, Jun 19, 2013 at 10:25 PM, Jason Rosenberg <j...@squareup.com> >> wrote: >> >> > In the 0.8 config, log.dir is now log.dirs. It looks like the singular >> > log.dir is still supported, but under the covers the property is >> log.dirs. >> > >> > I'm curious, does this take a comma separated list of directories? The >> new >> > config page just says: >> > "The directories in which the log data is kept" >> > >> > Also, how does kafka handle multiple directories? Does it treat each >> > directory as a separate replica partition, or what? >> > >> > Jason >> > >>
