comments from a RAS guy below...
Adam Lindsay wrote:
Kent Watsen wrote:
What are you *most* interested in for this server? Reliability?
Capacity? High Performance? Reading or writing? Large contiguous reads
or small seeks?
One thing that I did that got a good feedback from this list was
picking apart the requirements of the most demanding workflow I
imagined for the machine I was speccing out.
My first posting contained my use-cases, but I'd say that video
recording/serving will dominate the disk utilization - thats why I'm
pushing for 4 striped sets of RAIDZ2 - I think that it would be all
around goodness
It sounds good, that way, but (in theory), you'll see random I/O suffer
a bit when using RAID-Z2: the extra parity will drag performance down a
bit. The RAS guys will flinch at this, but have you considered 8*(2+1)
RAID-Z1?
Nit: small, random read I/O may suffer. Large random read or any random
write workloads should be ok.
For 24 data disks there are enough combinations that it is not easy to
pick from. The attached RAIDoptimizer output may help you decide on
the trade-offs. For description of the theory behind it, see my blog
http://blogs.sun.com/relling
I recommend loading it into StarOffice and using graphs or sorts to
reorder the data, based on your priorities. Also, this uses a generic
model, knowing the drive model will allow bandwidth analysis (with the
caveats shown in Adam's blog below).
I don't want to over-pimp my links, but I do think my blogged
experiences with my server (also linked in another thread) might give
you something to think about:
http://lindsay.at/blog/archive/tag/zfs-performance/
I'm learning more and more about this subject as I test the server
(not all that dissimilar to what you've described, except with only 18
disks) I now have. I'm frustrated at the relative unavailability of
PCIe SATA controller cards that are ZFS-friendly (i.e., JBOD), and the
relative unavailability of motherboards that support both the latest
CPUs as well as have a good PCI-X architecture.
Good point - another reply I just sent noted a PCI-X sata controller
card, but I'd prefer a PCIe card - do you have a recommendation on a
PCIe card?
Yes, I (obviously :-) recommend
http://www.sun.com/storagetek/storage_networking/hba/sas/specs.xml
Note: marketing still seems to have SATA-phobia, so if you search for SATA
you'll be less successful than searching for SAS. But many SAS HBAs support
SATA devices, too.
-- richard
RAIDOptimizer Report
Date: Sep 14, 2007 at 9:50:42 AM
RAIDOptimizer version: 1.0
RAID Type Disks/Set Sets Spares Copies Space (GBytes)
MTTDL[1] (yrs) MTTDL[2] (yrs) Performance (iops) Min BW (MByte/s)
Max BW (MBytes/s) MTBS[1] (yrs) MTBS[2] (yrs) Note
RAID-Z 12 2 0 1 11,000 8,559 17 141 0
0 14 14 1
RAID-Z 8 3 0 1 10,500 13,450 27 212 0
0 14 14 1
RAID-Z 6 4 0 1 10,000 18,830 38 282 0
0 14 14 1
RAID-Z2 12 2 0 1 10,000 27,905,578 9,825 141
0 0 37 37 1
RAID-Z 4 6 0 1 9,000 31,383 63 424 0
0 14 14 1
RAID-Z2 8 3 0 1 9,000 73,086,037 15,439 212
0 0 37 37 1
RAID-Z 3 8 0 1 8,000 47,074 95 565 0
0 14 14 1
RAID-Z2 6 4 0 1 8,000 153,480,678 33,774 282
0 0 37 37 1
Mirror 2 12 0 1 6,000 108,076 190 1,694 0
0 14 14 1
RAID-Z2 4 6 0 1 6,000 511,602,260 150,106 424
0 0 37 37 1
RAID-Z2 3 8 0 1 4,000 1,534,806,780 675,475 565
0 0 37 37 1
RAID-Z 11 2 2 1 10,000 39,851 21 141 0
0 14 137 1
RAID-Z2 11 2 2 1 9,000 1,700,290,577 79,701 141
0 0 37 642 1
Mirror 2 11 2 1 5,500 797,011 208 1,553 0
0 14 137 1
RAID-Z 7 3 3 1 9,000 69,580 36 212 0
0 14 642 1
RAID-Z2 7 3 3 1 7,500 5,343,770,385 185,548 212
0 0 37 3,739 1
RAID-Z 3 7 3 1 7,000 208,741 109 494 0
0 14 642 1
RAID-Z2 3 7 3 1 3,500 80,156,555,773 5,964,029
494 0 0 37 3,739 1
RAID-Z 10 2 4 1 9,000 48,706 25 141 0
0 14 3,739 1
RAID-Z 5 4 4 1 8,000 109,589 57 282 0
0 14 3,739 1
RAID-Z2 10 2 4 1 8,000 2,337,899,543 97,412 141
0 0 37 26,549 1
RAID-Z 4 5 4 1 7,500 146,119 76 353 0
0 14 3,739 1
RAID-Z2 5 4 4 1 6,000 14,027,397,260 608,828 282
0 0 37 26,549 1
Mirror 2 10 4 1 5,000 876,712 228 1,412 0
0 14 3,739 1
RAID-Z2 4 5 4 1 5,000 28,054,794,521 1,461,187
353 0 0 37 26,549 1
RAID-Z 9 2 6 1 8,000 60,883 32 141 0
0 14 226,707 1
RAID-Z 6 3 6 1 7,500 97,412 51 212 0
0 14 226,707 1
RAID-Z2 9 2 6 1 7,000 3,339,856,491 124,251 141
0 0 37 2,303,989 1
RAID-Z 3 6 6 1 6,000 243,531 127 424 0
0 14 226,707 1
RAID-Z2 6 3 6 1 6,000 9,351,598,174 405,885 212
0 0 37 2,303,989 1
Mirror 2 9 6 1 4,500 974,125 254 1,271 0
0 14 226,707 1
RAID-Z2 3 6 6 1 3,000 93,515,981,735 8,117,707
424 0 0 37 2,303,989 1
RAID-Z 8 2 8 1 7,000 78,278 41 141 0
0 14 27,664,472 1
RAID-Z 4 4 8 1 6,000 182,648 95 282 0
0 14 27,664,472 1
RAID-Z2 8 2 8 1 6,000 5,009,784,736 217,439 141
0 0 37 390,579,088 1
Mirror 2 8 8 1 4,000 1,095,890 285 1,129
0 0 14 27,664,472 1
RAID-Z2 4 4 8 1 4,000 35,068,493,151 2,283,105
282 0 0 37 390,579,088 1
RAID-Z 5 3 9 1 6,000 146,119 76 212 0
0 14 390,579,088 1
RAID-Z 3 5 9 1 5,000 292,237 152 353 0
0 14 390,579,088 1
RAID-Z2 5 3 9 1 4,500 18,703,196,347 1,082,361
212 0 0 37 6,466,912,109 1
RAID-Z2 3 5 9 1 2,500 112,219,178,082 11,689,498
353 0 0 37 6,466,912,109 1
RAID-Z 7 2 10 1 6,000 104,371 54 141 0
0 14 6,466,912,109 1
RAID-Z2 7 2 10 1 5,000 8,015,655,577 417,482 141
0 0 37 125,488,655,903 1
Mirror 2 7 10 1 3,500 1,252,446 326 988
0 0 14 6,466,912,109 1
RAID-Z 6 2 12 1 5,000 146,119 76 141 0
0 14 2,857,251,381,405 1
RAID-Z 4 3 12 1 4,500 243,531 127 212 0
0 14 2,857,251,381,405 1
RAID-Z 3 4 12 1 4,000 365,297 190 282 0
0 14 2,857,251,381,405 1
RAID-Z2 6 2 12 1 4,000 14,027,397,260 913,242 141
0 0 37 76,332,197,074,076 1
Mirror 2 6 12 1 3,000 1,461,187 381 847
0 0 14 2,857,251,381,405 1
RAID-Z2 4 3 12 1 3,000 46,757,990,868 4,058,853
212 0 0 37 76,332,197,074,076 1
RAID-Z2 3 4 12 1 2,000 140,273,972,603 18,264,840
282 0 0 37 76,332,197,074,076 1
RAID-Z 5 2 14 1 4,000 219,178 114 141 0
0 14 114,155,251,141,552 1
RAID-Z2 5 2 14 1 3,000 28,054,794,521 2,435,312
141 0 0 37 114,155,251,141,552 1
Mirror 2 5 14 1 2,500 1,753,425 457 706
0 0 14 114,155,251,141,552 1
RAID-Z 3 3 15 1 3,000 487,062 254 212 0
0 14 114,155,251,141,552 1
RAID-Z2 3 3 15 1 1,500 187,031,963,470 32,470,827
212 0 0 37 114,155,251,141,552 1
RAID-Z 4 2 16 1 3,000 365,297 190 141 0
0 14 114,155,251,141,552 1
Mirror 2 4 16 1 2,000 2,191,781 571 565
0 0 14 114,155,251,141,552 1
RAID-Z2 4 2 16 1 2,000 70,136,986,301 9,132,420
141 0 0 37 114,155,251,141,552 1
RAID-Z 3 2 18 1 2,000 730,594 381 141 0
0 14 114,155,251,141,552 1
Mirror 2 3 18 1 1,500 2,922,374 761 424
0 0 14 114,155,251,141,552 1
RAID-Z2 3 2 18 1 1,000 280,547,945,205 73,059,361
141 0 0 37 114,155,251,141,552 1
Mirror 2 2 20 1 1,000 4,383,562 1,142 282
0 0 14 114,155,251,141,552 1
MTTDL[1] model uses MTBF and MTTR.
MTTDL[2] model also considers Unrecoverable Error Rate (UER)
Performance is for average, small, random, cache-miss, read I/O operations per
second (iops).
Minimum and maximum media read bandwidth assumes no caching, bandwidth limited
by aggregated media read speed.
Note 1:
RAID software: ZFS
Number of disks: 24
Usage: 50%
Copies: 1
Disk size: 500 GBytes
Disk MTBF: 800,000 hrs (1,250 FITs) (1.1% AFR)
Average read seek time: 10.0 ms
Rotational speed: 7200.0 RPM
Calculated small, random read performance: 70.59 iops
Max media bandwidth: 0.0 MBytes/s
Min media bandwidth: 0.0 MBytes/s
Unrecoverable Error Rate: 1.0E-14 per bits read
Logistical MTTR: 24 hrs
Mirror resynchronization rate: 60 GBytes/hr (17.07 MBytes/s)
RAID-Z resynchronization rate: 60 GBytes/hr (17.07 MBytes/s)
RAID-Z2 resynchronization rate: 60 GBytes/hr (17.07 MBytes/s)
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
