Thanks Jay, I'll do some testing with this and report back. Jason
On Thu, Aug 15, 2013 at 7:10 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > I believe either should work. The broker has a record of what it should > have in zk and will recreate any missing logs. Try it to make sure though. > > Sent from my iPhone > > On Aug 15, 2013, at 12:52 AM, Jason Rosenberg <j...@squareup.com> wrote: > > > Ok, that makes sense that the broker will shut itself down. > > > > If we bring it back up, can this be with an altered set of log.dirs? > Will > > the destroyed partitions get rebuilt on a new log.dir? Or do we have to > > bring it back up with a new or repaired disk, matching the old log.dir, > in > > order for those replicas to be rebuilt? > > > > Jason > > > > > > On Wed, Aug 14, 2013 at 4:16 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > >> If you get a disk error that results in an IOException the broker will > >> shut itself down. You would then have the option of replacing the disk > or > >> deleting that data directory from the list. When the broker is brought > back > >> up the intact partitions will quickly catch up and be online; the > destroyed > >> partitions will have to fully rebuild off the other replicas and will > take > >> a little longer but will automatically come back online once they have > >> restored off the replicas. > >> > >> -jay > >> > >> Sent from my iPhone > >> > >> On Aug 14, 2013, at 1:49 PM, Jason Rosenberg <j...@squareup.com> wrote: > >> > >>> I'm getting ready to try out this configuration (use multiple disks, no > >>> RAID, per broker). One concern is the procedure for recovering if > there > >> is > >>> a disk failure. > >>> > >>> If a disk fails, will the broker go offline, or will it continue > serving > >>> partitions on its remaining good disks? And if so, is there a > procedure > >>> for moving the partitions that were on the failed disk, but not > >> necessarily > >>> all the others on that broker? > >>> > >>> Jason > >>> > >>> > >>> On Thu, Jun 20, 2013 at 3:15 PM, Jason Rosenberg <j...@squareup.com> > >> wrote: > >>> > >>>> yeah, that would work! > >>>> > >>>> > >>>> On Thu, Jun 20, 2013 at 1:20 PM, Jay Kreps <jay.kr...@gmail.com> > wrote: > >>>> > >>>>> Yeah we didn't go as far as adding weighting or anything like that--I > >>>>> think we'd be open to a patch that did that as long as it was > >>>>> optional. In the short term you can obviously add multiple > directories > >>>>> on the same disk to increase its share. > >>>>> > >>>>> -Jay > >>>>> > >>>>> On Thu, Jun 20, 2013 at 12:59 PM, Jason Rosenberg <j...@squareup.com> > >>>>> wrote: > >>>>>> This sounds like a great idea, to just disks as "just a bunch of > >> disks" > >>>>> or > >>>>>> JBOD.....hdfs works well this way. > >>>>>> > >>>>>> Do all the disks need to be the same size, to use them evenly? > Since > >> it > >>>>>> will allocate partitions randomly? > >>>>>> > >>>>>> It would be nice if you had 2 disks, with one twice as large as the > >>>>> other, > >>>>>> if the larger would be twice as likely to receive partitions as the > >>>>> smaller > >>>>>> one, etc. > >>>>>> > >>>>>> I suppose this goes into my earlier question to the list, vis-a-vis > >>>>>> heterogeneous brokers (e.g. utilize brokers with different sized > >>>>> storage, > >>>>>> using some sort of weighting scheme, etc.). > >>>>>> > >>>>>> Jason > >>>>>> > >>>>>> > >>>>>> On Thu, Jun 20, 2013 at 11:07 AM, Jay Kreps <jay.kr...@gmail.com> > >>>>> wrote: > >>>>>> > >>>>>>> The intention is to allow the use of multiple disks without RAID or > >>>>>>> logical volume management. We have found that there are a lot of > >>>>>>> downsides to RAID--in particular a huge throughput hit. Since we > >>>>>>> already have a parallelism model due to partitioning and a fault > >>>>>>> tolerance model with replication RAID doesn't actually buy much. > With > >>>>>>> this feature you can directly mount multiple disks as their own > >>>>>>> directory and the server will randomly assign partitions to them. > >>>>>>> > >>>>>>> Obviously this will only work well if there are enough > >> high-throughput > >>>>>>> partitions to make load balance evenly (e.g. if you have only one > big > >>>>>>> partition per server then this isn't going to work). > >>>>>>> > >>>>>>> -Jay > >>>>>>> > >>>>>>> On Wed, Jun 19, 2013 at 11:01 PM, Jason Rosenberg < > j...@squareup.com> > >>>>>>> wrote: > >>>>>>>> is it possible for a partition to have multiple replicas on > >> different > >>>>>>>> directories on the same broker? (hopefully no!) > >>>>>>>> > >>>>>>>> > >>>>>>>> On Wed, Jun 19, 2013 at 10:47 PM, Jun Rao <jun...@gmail.com> > wrote: > >>>>>>>> > >>>>>>>>> It takes a comma separated list and partition replicas are > randomly > >>>>>>>>> distributed to the list. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> > >>>>>>>>> Jun > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Wed, Jun 19, 2013 at 10:25 PM, Jason Rosenberg < > >> j...@squareup.com > >>>>>> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> In the 0.8 config, log.dir is now log.dirs. It looks like the > >>>>>>> singular > >>>>>>>>>> log.dir is still supported, but under the covers the property is > >>>>>>>>> log.dirs. > >>>>>>>>>> > >>>>>>>>>> I'm curious, does this take a comma separated list of > directories? > >>>>>>> The > >>>>>>>>> new > >>>>>>>>>> config page just says: > >>>>>>>>>> "The directories in which the log data is kept" > >>>>>>>>>> > >>>>>>>>>> Also, how does kafka handle multiple directories? Does it treat > >>>>> each > >>>>>>>>>> directory as a separate replica partition, or what? > >>>>>>>>>> > >>>>>>>>>> Jason > >> >
