Wow Craig - thank you so much for that thorough response. I am only using 1 vdev and I didn't realize two things:
1) that 50 GB on each of the 300s is essentially wasted. I thought it would spread 300 GB of parity across all 6 disks, leaving me with 1350 GB of "data" space. Instead, you're saying 250 GB of parity is spread across all 6 disks and an additional 150 GB is ignored, leaving me with, as you said, 1250 GB "data" space. 2) I wasn't sure if zpool list described the total before or after parity was subtracted (and "extra" space ignored). Thanks for clearing that up. However, I'm still a little confused how this adds up to 1.11T and 1.36T for zfs list and zpool list, respectively (my box is under construction atm so I can't capture the exact output). To minimize the wasted space, can I create 1x250 and 1x50 GB partitions on each of the 300 GB drives, then make them a new raidz vdev? 1st raidz 6x250 GB partitions: 1250 GB data space 2nd raidz 3x50 GB partitions: 100 GB data space total: 1350 GB data space Can I pool these together into one large pool? I'm trying to think it if would be possible to lose data based on any one disk failure. As my signature says, despite working for Sun I work on everything *but* Solaris every day, so I really appreciate the guidance. Just an FYI - I started with opensolaris but I needed to add a combo IDE/SATA card to handle all my spare disks (mostly varying sizes) - which wasn't recognized by opensolaris. I moved to Linux (which detects the drives on the I/O card) with zfs-fuse (filesystem in userspace - steps around CDDL/GPL incompatibility) but I found the features limited and where it compiled on one distribution that had its own set of unrelated issues, it failed to compile on another. Currently, I'm installing OpenSolaris in a VirtualBox VM on a Linux host using raw disk passthrough so I can use zfs with this I/O card. We'll see how it goes :) Thanks again, -Brent On Tue, 2009-03-17 at 18:02 -0700, Craig Cory wrote: > Brent, > > > Brent Wagner wrote: > > Can someone point me to a document describing how available space in a > > zfs is calculated or review the data below and tell me what I'm > > missing? > > > > Thanks in advance, > > -Brent > > === > > I have a home project with 3x250 GB+3x300 GB in raidz, so I expect to > > lose 1x300 GB to parity. > > > > Total size:1650GB > > Total size using 1024 to measure: ~1534 GB > > > > Expected raidz zpool size after losing 300 GB to parity: ~1350 GB > > Expected raidz zpool size using 1024 to measure: ~1255.5 GB > > > > Actual zpool size: 1.36T > > > > Single zfs on the pool - available size: 1.11T > > > > I realize zfs is going to have some overhead but 250 GB seems a little > > excessive...right? I thought maybe the zpool was showing all 6 disks and > > the filesystem reflected the remaining space after discounting the > > parity disk but that doesn't add up in a way that makes sense either > > (see above). Can someone help explain these numbers? > > > > Thanks, > > -Brent > > > > > When you say "3x250 GB+3x300 GB in raidz" do you mean: > > 1) # zpool create mypool raidz 250gb-1 250gb-2 250gb-3 300gb-1 \ > > 300gb-2 300gb-3 > > or > > 2) # zpool create mypool raidz 250gb-1 250gb-2 250gb-3 \ > > raidz 300gb-1 300gb-2 300gb-3 > > As I understand it, #1 would waste the extra 50gb on each 300gb drive and give > you 1500gb usable space. 250gb of that (1/6th) would be parity, so 1250gb > "data" space. > > #2 would make 2 vdevs of 750gb and 900gb totaling 1650gb space. Parity would > use 250gb from the 1st vdev and 300gb from the second; so 1100gb of "data" > space is available. > > Either way, when you list raidz* pools with > # zpool list > you see the total physical space. When you list the filesystems with > # zfs list > you get the usable filesystem space, which is where the parity is implemented. > > Here's an example with 250MB files and 300MB files: > > For #1 scenario: > > # zpool create -f mypool1 raidz /250d1 /250d2 /250d3 /300d1 /300d2 /300d3 > > # zpool list > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > mypool1 1.44G 145K 1.44G 0% ONLINE - > > # zfs list > NAME USED AVAIL REFER MOUNTPOINT > mypool1 115K 1.16G 40.7K /mypool1 > > # zpool status > pool: mypool1 > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > mypool1 ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > /250d1 ONLINE 0 0 0 > /250d2 ONLINE 0 0 0 > /250d3 ONLINE 0 0 0 > /300d1 ONLINE 0 0 0 > /300d2 ONLINE 0 0 0 > /300d3 ONLINE 0 0 0 > > ------------------------------------------------------------------------------ > And for #2: > # zpool create -f mypool2 raidz /250d1 /250d2 /250d3 raidz /300d1 /300d2 > /300d3 > > # zpool list > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > mypool2 1.58G 157K 1.58G 0% ONLINE - > > # zfs list > NAME USED AVAIL REFER MOUNTPOINT > mypool2 101K 1.02G 32.6K /mypool2 > > # zpool status > pool: mypool2 > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > mypool2 ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > /250d1 ONLINE 0 0 0 > /250d2 ONLINE 0 0 0 > /250d3 ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > /300d1 ONLINE 0 0 0 > /300d2 ONLINE 0 0 0 > /300d3 ONLINE 0 0 0 > > errors: No known data errors > ------------------------------------------------------------------- > > Does this describe what you're seeing? > > > Craig > > > > -- > Craig Cory > Senior Instructor :: ExitCertified > : Sun Certified System Administrator > : Sun Certified Network Administrator > : Sun Certified Security Administrator > : Veritas Certified Instructor > > 8950 Cal Center Drive > Bldg 1, Suite 110 > Sacramento, California 95826 > [e] craig.c...@exitcertified.com > [p] 916.669.3970 > [f] 916.669.3977 > [w] WWW.EXITCERTIFIED.COM > +---------------------------------------------------------------------+ > OTTAWA | SACRAMENTO | MONTREAL | LAS VEGAS | QUEBEC CITY | CALGARY > SAN FRANCISCO | VANCOUVER | REGINA | WINNIPEG | TORONTO > -- Brent Wagner Support Engineer - Windows/Linux/VMware Sun Microsystems, Inc. _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
