On Wed, Sep 8, 2010 at 15:27, Edward Ned Harvey <sh...@nedharvey.com> wrote: >> From: pantz...@gmail.com [mailto:pantz...@gmail.com] On Behalf Of >> Mattias Pantzare >> >> It >> is about 1 vdev with 12 disk or 2 vdev with 6 disks. If you have 2 >> vdev you have to read half the data compared to 1 vdev to resilver a >> disk. > > Let's suppose you have 1T of data. You have 12-disk raidz2. So you have > approx 100G on each disk, and you replace one disk. Then 11 disks will each > read 100G, and the new disk will write 100G. > > Let's suppose you have 1T of data. You have 2 vdev's that are each 6-disk > raidz1. Then we'll estimate 500G is on each vdev, so each disk has approx > 100G. You replace a disk. Then 5 disks will each read 100G, and 1 disk > will write 100G. > > Both of the above situations resilver in equal time, unless there is a bus > bottleneck. 21 disks in a single raidz3 will resilver just as fast as 7 > disks in a raidz1, as long as you are avoiding the bus bottleneck. But 21 > disks in a single raidz3 provides better redundancy than 3 vdev's each > containing a 7 disk raidz1. > > In my personal experience, approx 5 disks can max out approx 1 bus. (It > actually ranges from 2 to 7 disks, if you have an imbalance of cheap disks > on a good bus, or good disks on a crap bus, but generally speaking people > don't do that. Generally people get a good bus for good disks, and cheap > disks for crap bus, so approx 5 disks max out approx 1 bus.) > > In my personal experience, servers are generally built with a separate bus > for approx every 5-7 disk slots. So what it really comes down to is ... > > Instead of the Best Practices Guide saying "Don't put more than ___ disks > into a single vdev," the BPG should say "Avoid the bus bandwidth bottleneck > by constructing your vdev's using physical disks which are distributed > across multiple buses, as necessary per the speed of your disks and buses."
This is assuming that you have no other IO besides the scrub. You should of course keep the number of disks in a vdev low for general performance reasons unless you only have linear reads (as your IOPS will be close to what only one disk can give for the whole vdev). _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
