I believe either should work. The broker has a record of what it should have in zk and will recreate any missing logs. Try it to make sure though.
Sent from my iPhone On Aug 15, 2013, at 12:52 AM, Jason Rosenberg <j...@squareup.com> wrote: > Ok, that makes sense that the broker will shut itself down. > > If we bring it back up, can this be with an altered set of log.dirs? Will > the destroyed partitions get rebuilt on a new log.dir? Or do we have to > bring it back up with a new or repaired disk, matching the old log.dir, in > order for those replicas to be rebuilt? > > Jason > > > On Wed, Aug 14, 2013 at 4:16 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > >> If you get a disk error that results in an IOException the broker will >> shut itself down. You would then have the option of replacing the disk or >> deleting that data directory from the list. When the broker is brought back >> up the intact partitions will quickly catch up and be online; the destroyed >> partitions will have to fully rebuild off the other replicas and will take >> a little longer but will automatically come back online once they have >> restored off the replicas. >> >> -jay >> >> Sent from my iPhone >> >> On Aug 14, 2013, at 1:49 PM, Jason Rosenberg <j...@squareup.com> wrote: >> >>> I'm getting ready to try out this configuration (use multiple disks, no >>> RAID, per broker). One concern is the procedure for recovering if there >> is >>> a disk failure. >>> >>> If a disk fails, will the broker go offline, or will it continue serving >>> partitions on its remaining good disks? And if so, is there a procedure >>> for moving the partitions that were on the failed disk, but not >> necessarily >>> all the others on that broker? >>> >>> Jason >>> >>> >>> On Thu, Jun 20, 2013 at 3:15 PM, Jason Rosenberg <j...@squareup.com> >> wrote: >>> >>>> yeah, that would work! >>>> >>>> >>>> On Thu, Jun 20, 2013 at 1:20 PM, Jay Kreps <jay.kr...@gmail.com> wrote: >>>> >>>>> Yeah we didn't go as far as adding weighting or anything like that--I >>>>> think we'd be open to a patch that did that as long as it was >>>>> optional. In the short term you can obviously add multiple directories >>>>> on the same disk to increase its share. >>>>> >>>>> -Jay >>>>> >>>>> On Thu, Jun 20, 2013 at 12:59 PM, Jason Rosenberg <j...@squareup.com> >>>>> wrote: >>>>>> This sounds like a great idea, to just disks as "just a bunch of >> disks" >>>>> or >>>>>> JBOD.....hdfs works well this way. >>>>>> >>>>>> Do all the disks need to be the same size, to use them evenly? Since >> it >>>>>> will allocate partitions randomly? >>>>>> >>>>>> It would be nice if you had 2 disks, with one twice as large as the >>>>> other, >>>>>> if the larger would be twice as likely to receive partitions as the >>>>> smaller >>>>>> one, etc. >>>>>> >>>>>> I suppose this goes into my earlier question to the list, vis-a-vis >>>>>> heterogeneous brokers (e.g. utilize brokers with different sized >>>>> storage, >>>>>> using some sort of weighting scheme, etc.). >>>>>> >>>>>> Jason >>>>>> >>>>>> >>>>>> On Thu, Jun 20, 2013 at 11:07 AM, Jay Kreps <jay.kr...@gmail.com> >>>>> wrote: >>>>>> >>>>>>> The intention is to allow the use of multiple disks without RAID or >>>>>>> logical volume management. We have found that there are a lot of >>>>>>> downsides to RAID--in particular a huge throughput hit. Since we >>>>>>> already have a parallelism model due to partitioning and a fault >>>>>>> tolerance model with replication RAID doesn't actually buy much. With >>>>>>> this feature you can directly mount multiple disks as their own >>>>>>> directory and the server will randomly assign partitions to them. >>>>>>> >>>>>>> Obviously this will only work well if there are enough >> high-throughput >>>>>>> partitions to make load balance evenly (e.g. if you have only one big >>>>>>> partition per server then this isn't going to work). >>>>>>> >>>>>>> -Jay >>>>>>> >>>>>>> On Wed, Jun 19, 2013 at 11:01 PM, Jason Rosenberg <j...@squareup.com> >>>>>>> wrote: >>>>>>>> is it possible for a partition to have multiple replicas on >> different >>>>>>>> directories on the same broker? (hopefully no!) >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jun 19, 2013 at 10:47 PM, Jun Rao <jun...@gmail.com> wrote: >>>>>>>> >>>>>>>>> It takes a comma separated list and partition replicas are randomly >>>>>>>>> distributed to the list. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Jun >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jun 19, 2013 at 10:25 PM, Jason Rosenberg < >> j...@squareup.com >>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> In the 0.8 config, log.dir is now log.dirs. It looks like the >>>>>>> singular >>>>>>>>>> log.dir is still supported, but under the covers the property is >>>>>>>>> log.dirs. >>>>>>>>>> >>>>>>>>>> I'm curious, does this take a comma separated list of directories? >>>>>>> The >>>>>>>>> new >>>>>>>>>> config page just says: >>>>>>>>>> "The directories in which the log data is kept" >>>>>>>>>> >>>>>>>>>> Also, how does kafka handle multiple directories? Does it treat >>>>> each >>>>>>>>>> directory as a separate replica partition, or what? >>>>>>>>>> >>>>>>>>>> Jason >>
