yeah, that would work!
On Thu, Jun 20, 2013 at 1:20 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > Yeah we didn't go as far as adding weighting or anything like that--I > think we'd be open to a patch that did that as long as it was > optional. In the short term you can obviously add multiple directories > on the same disk to increase its share. > > -Jay > > On Thu, Jun 20, 2013 at 12:59 PM, Jason Rosenberg <j...@squareup.com> > wrote: > > This sounds like a great idea, to just disks as "just a bunch of disks" > or > > JBOD.....hdfs works well this way. > > > > Do all the disks need to be the same size, to use them evenly? Since it > > will allocate partitions randomly? > > > > It would be nice if you had 2 disks, with one twice as large as the > other, > > if the larger would be twice as likely to receive partitions as the > smaller > > one, etc. > > > > I suppose this goes into my earlier question to the list, vis-a-vis > > heterogeneous brokers (e.g. utilize brokers with different sized storage, > > using some sort of weighting scheme, etc.). > > > > Jason > > > > > > On Thu, Jun 20, 2013 at 11:07 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > >> The intention is to allow the use of multiple disks without RAID or > >> logical volume management. We have found that there are a lot of > >> downsides to RAID--in particular a huge throughput hit. Since we > >> already have a parallelism model due to partitioning and a fault > >> tolerance model with replication RAID doesn't actually buy much. With > >> this feature you can directly mount multiple disks as their own > >> directory and the server will randomly assign partitions to them. > >> > >> Obviously this will only work well if there are enough high-throughput > >> partitions to make load balance evenly (e.g. if you have only one big > >> partition per server then this isn't going to work). > >> > >> -Jay > >> > >> On Wed, Jun 19, 2013 at 11:01 PM, Jason Rosenberg <j...@squareup.com> > >> wrote: > >> > is it possible for a partition to have multiple replicas on different > >> > directories on the same broker? (hopefully no!) > >> > > >> > > >> > On Wed, Jun 19, 2013 at 10:47 PM, Jun Rao <jun...@gmail.com> wrote: > >> > > >> >> It takes a comma separated list and partition replicas are randomly > >> >> distributed to the list. > >> >> > >> >> Thanks, > >> >> > >> >> Jun > >> >> > >> >> > >> >> On Wed, Jun 19, 2013 at 10:25 PM, Jason Rosenberg <j...@squareup.com> > >> >> wrote: > >> >> > >> >> > In the 0.8 config, log.dir is now log.dirs. It looks like the > >> singular > >> >> > log.dir is still supported, but under the covers the property is > >> >> log.dirs. > >> >> > > >> >> > I'm curious, does this take a comma separated list of directories? > >> The > >> >> new > >> >> > config page just says: > >> >> > "The directories in which the log data is kept" > >> >> > > >> >> > Also, how does kafka handle multiple directories? Does it treat > each > >> >> > directory as a separate replica partition, or what? > >> >> > > >> >> > Jason > >> >> > > >> >> > >> >
