[zfs-discuss] ZFS mirror to RAIDz?
Hi all, I currently have four drives in my OpenSolaris box. The drives are split into two mirrors, one mirror containing my rpool (disks 1 & 2) and one containing other data (disks 2 & 3). I'm running out of space on my data mirror and am thinking of upgrading it to two 2TB disks. I then considered replacing disk 2 with a 2TB disk and making a RAIDz from the three new drives. I know this would leave my rpool vulnerable to hard drive failure, but I've got no data on it that can't be replaced with a reinstall. Can this be done easily? Or will I have to transfer all of my data to another machine and build the RAIDz from scratch, then transfer the data back? Thanks for any advice, Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS compression
Hi all, I'm running out of space on my OpenSolaris file server and can't afford to buy any new storage for a short while. Seeing at the machine has a dual core CPU at 2.2GHz and 4GB ram, I was thinking compression might be the way to go... I've read a small amount about compression, enough to find that it'll effect performance (not a problem for me) and that once you enable compression it only effects new files written to the file system. Is this still true of b134? And if it is, how can I compress all of the current data on the file system? Do I have to move it off then back on? Thanks for any advice, Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS compression
Thanks Alex, I've set compression on and have transferred data from the OpenSolaris machine to my Mac, deleted any snapshots and am now transferring them back. It seems to be working, but there's lots to transfer! I didn't know that MacZFS was still going, it's great to hear that people are still working on it. I may have to pluck up the courage to put it on my Mac Pro if I do a rebuild anytime soon. Thanks again, Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Increase size of ZFS mirror
Hi all, I have a ZFS mirror of two 500GB disks, I'd like to up these to 1TB disks, how can I do this? I must break the mirror as I don't have enough controller on my system board. My current mirror looks like this: [b]r...@beleg-ia:/share/media# zpool status share pool: share state: ONLINE scrub: none requested config: NAMESTATE READ WRITE CKSUM share ONLINE 0 0 0 mirrorONLINE 0 0 0 c5d0s0 ONLINE 0 0 0 c5d1s0 ONLINE 0 0 0 errors: No known data errors[/b] If I detach c5d1s0, add a 1TB drive, attach that, wait for it to resilver, then detach c5d0s0 and add another 1TB drive and attach that to the zpool, will that up the storage of the pool? Thanks very much, Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Increase size of ZFS mirror
Thomas, Could you post an example of what you mean (ie commands in the order to use them)? I've not played with ZFS that much and I don't want to muck my system up (I have data backed up, but am more concerned about getting myself in a mess and having to reinstall, thus losing my configurations). Many thanks for both of your replies, Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Increase size of ZFS mirror
Many thanks Thomas, I have a test machine so I shall try it on that before I try it on my main system. Thanks very much once again, Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Increase size of ZFS mirror
Thanks very much everyone. Victor, I did think about using VirtualBox, but I have a real machine and a supply of hard drives for a short time, for I'll test it out using that if I can. Scott, of course, at work we use three mirrors and it works very well, has saved us on occasion where we have detached the third mirror, upgraded, found the upgrade failed and have been able to revert from the third mirror instead of having to go through backups. George, it will be great to see the 'autoexpand' in the next release. I'm keeping my home server on stable releases for the time being :) -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS mirror and sun STK 2540 FC array
Hi all,
we have just bought a sun X2200M2 (4GB / 2 opteron 2214 / 2 disks 250GB
SATA2, solaris 10 update 4)
and a sun STK 2540 FC array (8 disks SAS 146 GB, 1 raid controller).
The server is attached to the array with a single 4 Gb Fibre Channel link.
I want to make a mirror using ZFS with this array.
I have created 2 volumes on the array
in RAID0 (stripe of 128 KB) presented to the host with lun0 and lun1.
So, on the host :
bash-3.00# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1d0
/[EMAIL PROTECTED],0/[EMAIL PROTECTED]/[EMAIL PROTECTED]/[EMAIL
PROTECTED],0
1. c2d0
/[EMAIL PROTECTED],0/[EMAIL PROTECTED]/[EMAIL PROTECTED]/[EMAIL
PROTECTED],0
2. c6t600A0B800038AFBC02F7472155C0d0
/scsi_vhci/[EMAIL PROTECTED]
3. c6t600A0B800038AFBC02F347215518d0
/scsi_vhci/[EMAIL PROTECTED]
Specify disk (enter its number):
bash-3.00# zpool create tank mirror
c6t600A0B800038AFBC02F347215518d0 c6t600A0B800038AFBC02F7472155C0d0
bash-3.00# df -h /tank
Filesystem size used avail capacity Mounted on
tank 532G24K 532G 1%/tank
I have tested the performance with a simple dd
[
time dd if=/dev/zero of=/tank/testfile bs=1024k count=1
time dd if=/tank/testfile of=/dev/null bs=1024k count=1
]
command and it gives :
# local throughput
stk2540
mirror zfs /tank
read 232 MB/s
write 175 MB/s
# just to test the max perf I did:
zpool destroy -f tank
zpool create -f pool c6t600A0B800038AFBC02F347215518d0
And the same basic dd gives me :
single zfs /pool
read 320 MB/s
write 263 MB/s
Just to give an idea the SVM mirror using the two local sata2 disks
gives :
read 58 MB/s
write 52 MB/s
So, in production the zfs /tank mirror will be used to hold
our home directories (10 users using 10GB each),
our projects files (200 GB mostly text files and cvs database),
and some vendors tools (100 GB).
People will access the data (/tank) using nfs4 with their
workstations (sun ultra 20M2 with centos 4update5).
On the ultra20 M2, the basic test via nfs4 gives :
read 104 MB/s
write 63 MB/s
A this point, I have the following questions :
-- Does someone has some similar figures about the STK 2540 using zfs ?
-- Instead of doing only 2 volumes in the array,
what do you think about doing 8 volumes (one for each disk)
and doing a 4 two way mirror :
zpool create tank mirror c6t6001.. c6t6002.. mirror c6t6003..
c6t6004.. {...} mirror c6t6007.. c6t6008..
-- I will add 4 disks in the array next summer.
Do you think I should create 2 new luns in the array
and doing a :
zpool add tank mirror c6t6001..(lun3) c6t6001..(lun4)
or build from scratch the 2 luns (6 disks raid0) , and the pool tank
(ie : backup /tank - zpool destroy -- add disk - reconfigure array
-- zpool create tank ... - restore backuped data)
-- I think about doing a disk scrubbing once a month.
Is it sufficient ?
-- Have you got any comment on the performance from the nfs4 client ?
If you add any advices / suggestions, feel free to share.
Thanks,
Benjamin
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] offlining a storage pool
Hi, I would like to offline an entire storage pool (not some devices), ( I want to stop all io activity to the pool) Maybe it could be implemented with a a command like : zpool offline -f tank which should implicity do a zfs unmount tank I use zfs with solaris 10 update 4. Thanks, Ben ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] zfs 32bits
Hi, I know that is not recommended by Sun to use ZFS on 32 bits machines but, what are really the consequences of doing this ? I have an old Bipro Xeon server (6 GB ram , 6 disks), and I would like to do a raidz with 4 disks with Solaris 10 update 4. Thanks, Ben ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Benchmarking Methodologies
I'm doing a little research study on ZFS benchmarking and performance profiling. Like most, I've had my favorite methods, but I'm re-evaluating my choices and trying to be a bit more scientific than I have in the past. To that end, I'm curious if folks wouldn't mind sharing their work on the subject? What tool(s) to you prefer in what situations? Do you have a standard method of running them (tool args; block sizes, thread counts, ...) or procedures between runs (zpool import/export, new dataset creation,...)? etc. Any feedback is appreciated. I want to get a good sampling of opinions. Thanks! benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Benchmarking Methodologies
On 4/21/10 2:15 AM, Robert Milkowski wrote: > I haven't heard from you in a while! Good to see you here again :) > > Sorry for stating obvious but at the end of a day it depends on what > your goals are. > Are you interested in micro-benchmarks and comparison to other file > systems? > > I think the most relevant filesystem benchmarks for users is when you > benchmark a specific application and present results from an > application point of view. For example, given a workload for Oracle, > MySQL, LDAP, ... how quickly it completes? How much benefit there is > by using SSDs? What about other filesystems? > > Micro-benchmarks are fine but very hard to be properly interpreted by > most users. > > Additionally most benchmarks are almost useless if they are not > compared to some other configuration with only a benchmarked component > changed. For example, knowing that some MySQL load completes in 1h on > ZFS is basically useless. But knowing that on the same HW with > Linux/ext3 and under the same load it completes in 2h would be > interesting to users. > > Other interesting thing would be to see an impact of different ZFS > setting on a benchmark results (aligned recordsize for database vs. > default, atime off vs. on, lzjb, gzip, ssd). Also comparison of > benchmark results with all default zfs setting compared to whatever > setting you did which gave you the best result. Hey Robert... I'm always around. :) You've made an excellent case for benchmarking and where its useful but what I'm asking for on this thread is for folks to share the research they've done with as much specificity as possible for research purposes. :) Let me illustrate: To Darren's point on FileBench and vdbench... to date I've found these two to be the most useful. IOzone, while very popular, has always given me strange results which are inconsistent regardless of how large the block and data is. Given that the most important aspect of any benchmark is repeatability and sanity in results, I've found no value in IOzone any longer. vdbench has become my friend particularly in the area of physical disk profiling. Before tuning ZFS (or any filesystem) its important to find a solid baseline of performance on the underlying disk structure. So using a variety of vdbench profiles such as the following help you pinpoint exactly the edges of the performance envelope: sd=sd1,lun=/dev/rdsk/c0t1d0s0,threads=1 wd=wd1,sd=sd1,readpct=100,rhpct=0,seekpct=0 rd=run1,wd=wd1,iorate=max,elapsed=10,interval=1,forxfersize=(4k-4096k,d) With vdbench and the workload above I can get consistent, reliable results time after time and the results on other systems match. This is particularly key if your running a hardware RAID controller under ZFS. There isn't anything dd can do that vdbench can't do better. Using a workload like above both at differing xfer sizes and also at differing thread counts really helps give an accurate picture of the disk capabilities. Moving up into the filesystem. I've been looking intently at improving my FileBench profiles, based on the supplied ones with tweaking. I'm trying to get to a methodology that provides me with time-after-time repeatable results for real comparison between systems. I'm looking hard at vdbench file workloads, but they aren't yet nearly as sophisticated as FileBench. I am also looking at FIO (http://freshmeat.net/projects/fio/), which is FileBench-esce. At the end of the day, I agree entirely that application benchmarks are far more effective judges... but they are also more time consuming and less flexible than dedicated tools. The key is honing generic benchmarks to provide useful data which can be relied upon for making accurate estimates as regards to application performance. When you start judging filesystem performance based on something like MySQL there are simply too many variables involved. So, I appreciate the Benchmark 101, but I'm looking for anyone interested in sharing meat. Most of the existing ZFS benchmarks folks published are several years old now, and most were using IOzone. benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Plugging in a hard drive after Solaris has booted up?
On 5/7/10 9:38 PM, Giovanni wrote: > Hi guys, > > I have a quick question, I am playing around with ZFS and here's what I did. > > I created a storage pool with several drives. I unplugged 3 out of 5 drives > from the array, currently: > > NAMESTATE READ WRITE CKSUM > gpool UNAVAIL 0 0 0 insufficient replicas > raidz1UNAVAIL 0 0 0 insufficient replicas > c8t2d0 UNAVAIL 0 0 0 cannot open > c8t4d0 UNAVAIL 0 0 0 cannot open > c8t0d0 UNAVAIL 0 0 0 cannot open > > These drives had power all the time, the SATA cable however was disconnected. > Now, after I logged into Solaris and opened firefox, I plugged them back in > to sit and watch if the storage pool suddenly becomes "available" > > This did not happen, so my question is, do I need to make Solaris re-detect > the hard drives and if so how? I tried format -e but it did not seem to > detect the 3 drives I just plugged back in. Is this a BIOS issue? > > Does hot-swap hard drives only work when you replace current hard drives > (previously detected by BIOS) with others but not when you have ZFS/Solaris > running and want to add more storage without shutting down? > > It all boils down to, say the scenario is that I will need to purchase more > hard drives as my array grows, I would like to be able to (without shutting > down) add the drives to the storage pool (zpool) > There are lots of different things you can look at and do, but it comes down to just one command: "devfsadm -vC". This will cleanup (-C for cleanup, -v for verbose) the device tree if it gets into a funky state. Then run "format" or "iostat -En" to verify that the device(s) are there. Then re-import the zpool or add the device or whatever you wish to do. Even if device locations change, ZFS will do the right thing on import. If you wish to dig deeper... normally when you attach a new device hot-plug will do the right thing and you'll see the connection messages in "dmesg". If you want to explicitly check the state of dynamic reconfiguration, checkout the "cfgadm" command. Normally, however, on modern version of Solaris there is no reason to resort to that, its just something fun if you wish to dig. benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Mirrored Servers
On 5/8/10 3:07 PM, Tony wrote: > Lets say I have two servers, both running opensolaris with ZFS. I basically > want to be able to create a filesystem where the two servers have a common > volume, that is mirrored between the two. Meaning, each server keeps an > identical, real time backup of the other's data directory. Set them both up > as file servers, and load balance between the two for incoming requests. > > How would anyone suggest doing this? > I would carefully consider whether or not the _really_ need to be real time. Can you tolerate 5 minutes or even just 60 seconds of difference between them? If you can, then things are much easier and less complex. I'd personally use ZFS Snapshots to keep the two servers in sync every 60 seconds. As for load balancing, that depends on which protocal your using. FTP is easy. NFS/CIFS is a little harder. I'd simply use a load balancer (Zeus, NetScaler, Balance, HA-Proxy, etc.), but that is a little scary and bizarre in the case of NFS/CIFS, where you should instead use a single-server failover solution, such as Sun Cluster. benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS Hard disk buffer at 100%
The drive (c7t2d0)is bad and should be replaced. The second drive (c7t5d0) is either bad or going bad. This is exactly the kind of problem that can force a Thumper to it knees, ZFS performance is horrific, and as soon as you drop the bad disks things magicly return to normal. My first recommendation is to pull the SMART data from the disks if you can. I wrote a blog entry about SMART to address exactly the behavior your seeing back in 2008: http://www.cuddletech.com/blog/pivot/entry.php?id=993 Yes, people will claim that SMART data is useless for predicting failures, but in a case like yours you are just looking for data to corroborate a hypothesis. In order to test this condition, "zpool offline..." c7t2d0, which emulated removal. See if performance improves. On Thumpers I'd build a list of "suspect disks" based on 'iostat', like you show, and then correlate the SMART data, and then systematically offline disks to see if it really was the problem. In my experience the only other reason you'll legitimately see really wierd "bottoming out" of IO like this is if you hit the max conncurrent IO limits in ZFS (untill recently that limit was 35), so you'd see actv=35, and then when the device finally processed the IO's the thing would snap back to life. But even in those cases you shouldn't see request times (asvc_t) rise above 200ms. All that to say, replace those disks or at least test it. SSD's won't help, one or more drives are toast. benr. On 5/8/10 9:30 PM, Emily Grettel wrote: > Hi Giovani, > > Thanks for the reply. > > Here's a bit of iostat after uncompressing a 2.4Gb RAR file that has 1 > DWF file that we use. > > extended device statistics > r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 1.0 13.0 26.0 18.0 0.0 0.00.00.8 0 1 c7t1d0 > 2.05.0 77.0 12.0 2.4 1.0 343.8 142.8 100 100 c7t2d0 > 1.0 16.0 25.5 15.5 0.0 0.00.00.3 0 0 c7t3d0 > 0.0 10.00.0 17.0 0.0 0.03.21.2 1 1 c7t4d0 > 1.0 12.0 25.5 15.5 0.4 0.1 32.4 10.9 14 14 c7t5d0 > 1.0 15.0 25.5 18.0 0.0 0.00.10.1 0 0 c0t1d0 > extended device statistics > r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.00.00.00.0 2.0 1.00.00.0 100 100 c7t2d0 > 1.00.00.50.0 0.0 0.00.00.1 0 0 c7t0d0 > extended device statistics > r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 5.0 15.0 128.0 18.0 0.0 0.00.01.8 0 3 c7t1d0 > 1.09.0 25.5 18.0 2.0 1.8 199.7 179.4 100 100 c7t2d0 > 3.0 13.0 102.5 14.5 0.0 0.10.05.2 0 5 c7t3d0 > 3.0 11.0 102.0 16.5 0.0 0.12.34.2 1 6 c7t4d0 > 1.04.0 25.52.0 0.4 0.8 71.3 158.9 12 79 c7t5d0 > 5.0 16.0 128.5 19.0 0.0 0.10.12.6 0 5 c0t1d0 > extended device statistics > r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.04.00.02.0 2.0 2.0 496.1 498.0 99 100 c7t2d0 > 0.00.00.00.0 0.0 1.00.00.0 0 100 c7t5d0 > extended device statistics > r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 7.00.0 204.50.0 0.0 0.00.00.2 0 0 c7t1d0 > 1.00.0 25.50.0 3.0 1.0 2961.6 1000.0 99 100 c7t2d0 > 8.00.0 282.00.0 0.0 0.00.00.3 0 0 c7t3d0 > 6.00.0 282.50.0 0.0 0.06.12.3 1 1 c7t4d0 > 0.03.00.05.0 0.5 1.0 165.4 333.3 18 100 c7t5d0 > 7.00.0 204.50.0 0.0 0.00.01.6 0 1 c0t1d0 > 2.02.0 89.0 12.0 0.0 0.03.16.1 1 2 c3t0d0 > 0.02.00.0 12.0 0.0 0.00.00.2 0 0 c3t1d0 > > Sometimes two or more disks are going at 100. How does one solve this > issue if its a firmware bug? I tried looking around for Western > Digital Firmware for WD10EADS but couldn't find any available. > > Would adding an SSD or two help here? > > Thanks, > Em > > > Date: Fri, 7 May 2010 14:38:25 -0300 > Subject: Re: [zfs-discuss] ZFS Hard disk buffer at 100% > From: gtirl...@sysdroid.com > To: emilygrettelis...@hotmail.com > CC: zfs-discuss@opensolaris.org > > > On Fri, May 7, 2010 at 8:07 AM, Emily Grettel > mailto:emilygrettelis...@hotmail.com>> > wrote: > > Hi, > > I've had my RAIDz volume working well on SNV_131 but it has come > to my attention that there has been some read issues with the > drives. Previously I thought this was a CIFS problem but I'm > noticing that when transfering files or uncompressing some fairly > large 7z (1-2Gb) files (or even smaller rar - 200-300Mb) files > occasionally running iostat will give th
[zfs-discuss] Pool is wrong size in b134
I upgraded a server today that has been running SXCE b111 to the OpenSolaris preview b134. It has three pools and two are fine, but one comes up with no space available in the pool (SCSI jbod of 300GB disks). The zpool version is at 14. I tried exporting the pool and re-importing and I get several errors like this both exporting and importing: # zpool export pool1 WARNING: metaslab_free_dva(): bad DVA 0:645838978048 WARNING: metaslab_free_dva(): bad DVA 0:645843271168 ... I tried removing the zpool.cache file, rebooting, importing and receive no warnings, but still reporting the wrong avail and size. # zfs list pool1 NAMEUSED AVAIL REFER MOUNTPOINT pool1 396G 0 3.22M /export/home # zpool list pool1 NAMESIZE ALLOC FREECAP DEDUP HEALTH ALTROOT pool1 476G 341G 135G71% 1.00x ONLINE - # zpool status pool1 pool: pool1 state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scrub: none requested config: NAME STATE READ WRITE CKSUM pool1ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 c1t8d0 ONLINE 0 0 0 c1t9d0 ONLINE 0 0 0 c1t10d0 ONLINE 0 0 0 c1t11d0 ONLINE 0 0 0 c1t12d0 ONLINE 0 0 0 c1t13d0 ONLINE 0 0 0 c1t14d0 ONLINE 0 0 0 errors: No known data errors I try exporting and again get the metaslab_free_dva() warnings. Imported again with no warnings, but same numbers as above. If I try to remove files or truncate files I receive no free space errors. I reverted back to b111 and here is what the pool really looks like. # zfs list pool1 NAMEUSED AVAIL REFER MOUNTPOINT pool1 396G 970G 3.22M /export/home # zpool list pool1 NAMESIZE USED AVAILCAP HEALTH ALTROOT pool1 1.91T 557G 1.36T28% ONLINE - # zpool status pool1 pool: pool1 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM pool1ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c1t8d0 ONLINE 0 0 0 c1t9d0 ONLINE 0 0 0 c1t10d0 ONLINE 0 0 0 c1t11d0 ONLINE 0 0 0 c1t12d0 ONLINE 0 0 0 c1t13d0 ONLINE 0 0 0 c1t14d0 ONLINE 0 0 0 errors: No known data errors Also, the disks were replaced one at a time last year from 73GB to 300GB to increase the size of the pool. Any idea why the pool is showing up as the wrong size in b134 and have anything else to try? I don't want to upgrade the pool version yet and then not be able to revert back... thanks, Ben ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Pool is wrong size in b134
Cindy, The other two pools are 2 disk mirrors (rpool and another). Ben Cindy Swearingen wrote: Hi Ben, Any other details about this pool, like how it might be different from the other two pools on this system, might be helpful... I'm going to try to reproduce this problem. We'll be in touch. Thanks, Cindy On 06/17/10 07:02, Ben Miller wrote: I upgraded a server today that has been running SXCE b111 to the OpenSolaris preview b134. It has three pools and two are fine, but one comes up with no space available in the pool (SCSI jbod of 300GB disks). The zpool version is at 14. I tried exporting the pool and re-importing and I get several errors like this both exporting and importing: # zpool export pool1 WARNING: metaslab_free_dva(): bad DVA 0:645838978048 WARNING: metaslab_free_dva(): bad DVA 0:645843271168 ... I tried removing the zpool.cache file, rebooting, importing and receive no warnings, but still reporting the wrong avail and size. # zfs list pool1 NAMEUSED AVAIL REFER MOUNTPOINT pool1 396G 0 3.22M /export/home # zpool list pool1 NAMESIZE ALLOC FREECAP DEDUP HEALTH ALTROOT pool1 476G 341G 135G71% 1.00x ONLINE - # zpool status pool1 pool: pool1 state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scrub: none requested config: NAME STATE READ WRITE CKSUM pool1ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 c1t8d0 ONLINE 0 0 0 c1t9d0 ONLINE 0 0 0 c1t10d0 ONLINE 0 0 0 c1t11d0 ONLINE 0 0 0 c1t12d0 ONLINE 0 0 0 c1t13d0 ONLINE 0 0 0 c1t14d0 ONLINE 0 0 0 errors: No known data errors I try exporting and again get the metaslab_free_dva() warnings. Imported again with no warnings, but same numbers as above. If I try to remove files or truncate files I receive no free space errors. I reverted back to b111 and here is what the pool really looks like. # zfs list pool1 NAMEUSED AVAIL REFER MOUNTPOINT pool1 396G 970G 3.22M /export/home # zpool list pool1 NAMESIZE USED AVAILCAP HEALTH ALTROOT pool1 1.91T 557G 1.36T28% ONLINE - # zpool status pool1 pool: pool1 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM pool1ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c1t8d0 ONLINE 0 0 0 c1t9d0 ONLINE 0 0 0 c1t10d0 ONLINE 0 0 0 c1t11d0 ONLINE 0 0 0 c1t12d0 ONLINE 0 0 0 c1t13d0 ONLINE 0 0 0 c1t14d0 ONLINE 0 0 0 errors: No known data errors Also, the disks were replaced one at a time last year from 73GB to 300GB to increase the size of the pool. Any idea why the pool is showing up as the wrong size in b134 and have anything else to try? I don't want to upgrade the pool version yet and then not be able to revert back... thanks, Ben ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS on Ubuntu
How much of a difference is there in supporting applications in between Ubuntu and OpenSolaris? I was not considering Ubuntu until OpenSOlaris would not load onto my machine... Any info would be great. I have not been able to find any sort of comparison of ZFS on Ubuntu and OS. Thanks. (My current OS install troubleshoot thread - http://opensolaris.org/jive/thread.jspa?messageID=488193) -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on Ubuntu
What supporting applications are there on Ubuntu for RAIDZ? -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on Ubuntu
I tried to post this question on the Ubuntu forum. Within 30 minutes my post was on the second page of new posts... Yah. Im really not down with using Ubuntu on my server here. But I may be forced to. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Using a zvol from your rpool as zil for another zpool
> We have a server with a couple X-25E's and a bunch of > larger SATA > disks. > > To save space, we want to install Solaris 10 (our > install is only about > 1.4GB) to the X-25E's and use the remaining space on > the SSD's for ZIL > attached to a zpool created from the SATA drives. > > Currently we do this by installing the OS using > SVM+UFS (to mirror the > OS between the two SSD's) and then using the > remaining space on a slice > as ZIL for the larger SATA-based zpool. > > However, SVM+UFS is more annoying to work with as far > as LiveUpgrade is > concerned. We'd love to use a ZFS root, but that > requires that the > entire SSD be dedicated as an rpool leaving no space > for ZIL. Or does > it? For every system I have ever done zfs root on, it's always been a slice on a disk. As an example, we have an x4500 with 1TB disks. For that root config, we are planning on something like 150G on s0, and the rest on S3. s0 for the rpool, and s3 for the qpool. We didn't want to have to deal with issues around flashing a huge volume, as we found out with our other x4500 with 500GB disks. AFAIK, it's only non-rpool disks that use the "whole disk", and I doubt there's some sort of specific feature with an SSD, but I could be wrong. I like your idea of a reasonably sized root rpool and the rest used for the ZIL. But if you're going to do LU, you should probably take a good look at how much space you need for the clones and snapshots on the rpool Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Opensolaris is apparently dead
On 8/13/10 9:02 PM, "C. Bergström" wrote: > Erast wrote: >> >> >> On 08/13/2010 01:39 PM, Tim Cook wrote: >>> http://www.theregister.co.uk/2010/08/13/opensolaris_is_dead/ >>> >>> I'm a bit surprised at this development... Oracle really just doesn't >>> get it. The part that's most disturbing to me is the fact they >>> won't be >>> releasing nightly snapshots. It appears they've stopped Illumos in its >>> tracks before it really even got started (perhaps that explains the >>> timing of this press release) >> >> Wrong. Be patient, with the pace of current Illumos development it >> soon will have all the closed binaries liberated and ready to sync up >> with promised ON code drops as dictated by GPL and CDDL licenses. > Illumos is just a source tree at this point. You're delusional, > misinformed, or have some big wonderful secret if you believe you have > all the bases covered for a pure open source distribution though.. > > What's closed binaries liberated really mean to you? > > Does it mean >a. You copy over the binary libCrun and continue to use some > version of Sun Studio to build onnv-gate >b. You debug the problems with and start to use ancient gcc-3 (at > the probably expense of performance regressions which most people > would find unacceptable) >c. Your definition is narrow and has missed some closed binaries > > > I think it's great people are still hopeful, working hard and going to > steward this forward, but I wonder.. What pace are you referring to? > The last commit to illumos-gate was 6 days ago and you're already not > even keeping it in sync.. Can you even build it yet and if so where's > the binaries? Illumos is 2 weeks old. Lets cut it a little slack. :) benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Opensolaris is apparently dead
On 8/14/10 1:12 PM, Frank Cusack wrote: > > Wow, what leads you guys to even imagine that S11 wouldn't contain > comstar, etc.? *Of course* it will contain most of the bits that > are current today in OpenSolaris. That's a very good question actually. I would think that COMSTAR would stay because its used by the Fishworks appliance... however, COMSTAR is a competitive advantage for DIY storage solutions. Maybe they will rip it out of S11 and make it an add-on or something. That would suck. I guess the only real reason you can't yank COMSTAR is because its now the basis for iSCSI Target support. But again, there is nothing saying that Target support has to be part of the standard OS offering. Scary to think about. :) benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Liveupgrade'd to U8 and now can't boot previous U6 BE :(
Hi, As a related issue to this (specifically CR 6884728) - any ideas how I should go about removing the old BE? When I attempt to run ludelete I get the following: $ lustatus Boot Environment Is Active ActiveCanCopy Name Complete NowOn Reboot Delete Status -- -- - -- -- 10_05-09 yes no noyes- 10_10-09 yes yesyes no - $ ludelete 10_05-09 System has findroot enabled GRUB Checking if last BE on any disk... ERROR: cannot mount '/.alt.10_05-09/var': directory is not empty ERROR: cannot mount mount point device ERROR: failed to mount file system on ERROR: unmounting partially mounted boot environment file systems ERROR: No such file or directory: error unmounting ERROR: cannot mount boot environment by name <10_05-09> ERROR: Failed to mount BE <10_05-09>. ERROR: Failed to mount BE <10_05-09>. cat: cannot open /tmp/.lulib.luclb.dsk.2797.10_05-09 ERROR: This boot environment <10_05-09> is the last BE on the above disk. ERROR: Deleting this BE may make it impossible to boot from this disk. ERROR: However you may still boot solaris if you have BE(s) on other disks. ERROR: You *may* have to change boot-device order in the BIOS to accomplish this. ERROR: If you still want to delete this BE <10_05-09>, please use the force option (-f). Unable to delete boot environment. My zfs setup now shows this: NAME USED AVAIL REFER MOUNTPOINT rpool 11.4G 4.26G 39.5K /rpool rpool/ROOT9.15G 4.26G18K legacy rpool/ROOT/10_10-09 9.14G 4.26G 4.04G / rpool/ROOT/10_10...@10_10-09 2.39G - 4.10G - rpool/ROOT/10_10-09/var 2.71G 4.26G 1.18G /var rpool/ROOT/10_10-09/v...@10_10-09 1.53G - 2.11G - rpool/ROOT/s10x_u7wos_08 17.4M 4.26G 4.10G /.alt.10_05-09 rpool/ROOT/s10x_u7wos_08/var 9.05M 4.26G 2.11G /.alt.10_05-09/var rpool/dump1.00G 4.26G 1.00G - rpool/export 74.6M 4.26G19K /export rpool/export/home 74.5M 4.26G21K /export/home rpool/export/home/admin 65.5K 4.26G 65.5K /export/home/admin rpool/swap 1G 4.71G 560M - It seems that the ludelete script reassigns the mountpoint for the BE to be deleted - but falls foul of the /var mount underneath the old BE. I tried lumounting the old BE and checked the /etc/vfstab - but there are no extra zfs entries in there. I'm just looking for a clean way to remove the old BE, and then remove the old snapshot without interfering with Live Upgrade from working in the future. Many thanks, Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Liveupgrade'd to U8 and now can't boot previous U6 BE :(
> + dev=`echo $dev | sed 's/mirror.*/mirror/'` Thanks for the suggestion Kurt. However, I'm not running a mirror on that pool - so am guessing this won't help in my case. I'll try and pick my way through the lulib script if I get any time. Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] abusing zfs boot disk for fun and DR
I'm in the process of standing up a couple of t5440's, of which the config will eventually end up in another data center 6k miles from the original config, and I'm supposed to send disks to the data center and we'll start from there (yes, I know how to flar and jumpstart. When the boss says do something, sometimes you *just* have to do it) As I've already run into the boot failsafe when moving a root disk from one sparc host to another, I recently found out that a sys-unconfig'd disk does not suffer from the same problem. While I am probably going to be told, I shouldn't be doing this, I ran into an interesting "semantics" issue that I think zfs should at least be able to avoid (and which I have seen in other non-abusive configurations. ;-) 2 zfs disk, root mirrored. c2t0 and c2t1. hot unplug c2t0, (and I should probably have removed the busted mirror from c2t1, but I didn't) sys-unconfig disk in c2t1 move disk to new t5440 boot disk, and it enumerates everything correctly and then I notice zpool thinks it's degraded. I had added the mirror after I realized I wanted to run this by the list pool: rpool state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: resilver completed after 0h7m with 0 errors on Thu Jan 7 12:10:03 2010 config: NAME STATE READ WRITE CKSUM rpool DEGRADED 0 0 0 mirror DEGRADED 0 0 0 c2t0d0s0 ONLINE 0 0 0 c2t0d0s0 FAULTED 0 0 0 corrupted data c2t3d0s0 ONLINE 0 0 0 13.8G resilvered Anyway, should zfs report a faulted drive of the same ctd# which is already active? I understand why this happened, but from a logistics perspective, shouldn't zfs be smart enough to ignore a faulted disk like this? And this is not the first time I've had this scenario happen (I had an x4500 that had suffered through months of marvell driver bugs and corruption, and we probably had 2 or 3 of these types of things happen while trying to "soft" fix the problems). This also happened with hot-spares, which caused support to spend some time with back-line to figure out a procedure to clear those fauled disks which had the same ctd# as a working hot-spare... Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] abusing zfs boot disk for fun and DR
> Ben, > I have found that booting from cdrom and importing > the pool on the new host, then boot the hard disk > will prevent these issues. > That will reconfigure the zfs to use the new disk > device. > When running, zpool detach the missing mirror device > and attach a new one. Thanks. I'm well versed in dealing with zfs issues. The reason I reported this boot/rpool issue, was that it was similar in nature to issues that occured trying to remediate an x4500 which had suffered may sata disks go offline (due to the buggy Marvell driver) as well as corruption that occured while trying to fix said issue. Backline spent a fair amount of time just trying to remediate the issue with hot spares that looked exactly like the faulted config in my rpool. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Replacing a disk never completes
I have an X4540 running b134 where I'm replacing 500GB disks with 2TB disks (Seagate Constellation) and the pool seems sick now. The pool has four raidz2 vdevs (8+2) where the first set of 10 disks were replaced a few months ago. I replaced two disks in the second set (c2t0d0, c3t0d0) a couple of weeks ago, but have been unable to get the third disk to finish replacing (c4t0d0). I have tried the resilver for c4t0d0 four times now and the pool also comes up with checksum errors and a permanent error (:<0x0>). The first resilver was from 'zpool replace', which came up with checksum errors. I cleared the errors which triggered the second resilver (same result). I then did a 'zpool scrub' which started the third resilver and also identified three permanent errors (the two additional were in files in snapshots which I then destroyed). I then did a 'zpool clear' and then another scrub which started the fourth resilver attempt. This last attempt identified another file with errors in a snapshot that I have now destroyed. Any ideas how to get this disk finished being replaced without rebuilding the pool and restoring from backup? The pool is working, but is reporting as degraded and with checksum errors. Here is what the pool currently looks like: # zpool status -v pool2 pool: pool2 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver completed after 33h9m with 4 errors on Thu Sep 16 00:28:14 config: NAME STATE READ WRITE CKSUM pool2 DEGRADED 0 0 8 raidz2-0ONLINE 0 0 0 c0t4d0ONLINE 0 0 0 c1t4d0ONLINE 0 0 0 c2t4d0ONLINE 0 0 0 c3t4d0ONLINE 0 0 0 c4t4d0ONLINE 0 0 0 c5t4d0ONLINE 0 0 0 c2t5d0ONLINE 0 0 0 c3t5d0ONLINE 0 0 0 c4t5d0ONLINE 0 0 0 c5t5d0ONLINE 0 0 0 raidz2-1DEGRADED 0 014 c0t5d0ONLINE 0 0 0 c1t5d0ONLINE 0 0 0 c2t1d0ONLINE 0 0 0 c3t1d0ONLINE 0 0 0 c4t1d0ONLINE 0 0 0 c5t1d0ONLINE 0 0 0 c2t0d0ONLINE 0 0 0 c3t0d0ONLINE 0 0 0 replacing-8 DEGRADED 0 0 0 c4t0d0s0/o OFFLINE 0 0 0 c4t0d0 ONLINE 0 0 0 268G resilvered c5t0d0ONLINE 0 0 0 raidz2-2ONLINE 0 0 0 c0t6d0ONLINE 0 0 0 c1t6d0ONLINE 0 0 0 c2t6d0ONLINE 0 0 0 c3t6d0ONLINE 0 0 0 c4t6d0ONLINE 0 0 0 c5t6d0ONLINE 0 0 0 c2t7d0ONLINE 0 0 0 c3t7d0ONLINE 0 0 0 c4t7d0ONLINE 0 0 0 c5t7d0ONLINE 0 0 0 raidz2-3ONLINE 0 0 0 c0t7d0ONLINE 0 0 0 c1t7d0ONLINE 0 0 0 c2t3d0ONLINE 0 0 0 c3t3d0ONLINE 0 0 0 c4t3d0ONLINE 0 0 0 c5t3d0ONLINE 0 0 0 c2t2d0ONLINE 0 0 0 c3t2d0ONLINE 0 0 0 c4t2d0ONLINE 0 0 0 c5t2d0ONLINE 0 0 0 logs mirror-4ONLINE 0 0 0 c0t1d0s0 ONLINE 0 0 0 c1t3d0s0 ONLINE 0 0 0 cache c0t3d0s7ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: :<0x0> <0x167a2>:<0x552ed> (This second file was in a snapshot I destroyed after the resilver completed). # zpool list pool2 NAMESIZE ALLOC FREECAP DEDUP HEALTH ALTROOT pool2 31.8T 13.8T 17.9T43% 1.65x DEGRADED - The slog is a mirror of two SLC SSDs and the L2ARC is an MLC SSD. thanks, Ben _
Re: [zfs-discuss] Replacing a disk never completes
On 09/20/10 10:45 AM, Giovanni Tirloni wrote: On Thu, Sep 16, 2010 at 9:36 AM, Ben Miller mailto:bmil...@mail.eecis.udel.edu>> wrote: I have an X4540 running b134 where I'm replacing 500GB disks with 2TB disks (Seagate Constellation) and the pool seems sick now. The pool has four raidz2 vdevs (8+2) where the first set of 10 disks were replaced a few months ago. I replaced two disks in the second set (c2t0d0, c3t0d0) a couple of weeks ago, but have been unable to get the third disk to finish replacing (c4t0d0). I have tried the resilver for c4t0d0 four times now and the pool also comes up with checksum errors and a permanent error (:<0x0>). The first resilver was from 'zpool replace', which came up with checksum errors. I cleared the errors which triggered the second resilver (same result). I then did a 'zpool scrub' which started the third resilver and also identified three permanent errors (the two additional were in files in snapshots which I then destroyed). I then did a 'zpool clear' and then another scrub which started the fourth resilver attempt. This last attempt identified another file with errors in a snapshot that I have now destroyed. Any ideas how to get this disk finished being replaced without rebuilding the pool and restoring from backup? The pool is working, but is reporting as degraded and with checksum errors. [...] Try to run a `zpool clear pool2` and see if clears the errors. If not, you may have to detach `c4t0d0s0/o`. I believe it's a bug that was fixed in recent builds. I had tried a clear a few times with no luck. I just did a detach and that did remove the old disk and has now triggered another resilver which hopefully works. I had tried a remove rather than a detach before, but that doesn't work on raidz2... thanks, Ben -- Giovanni Tirloni gtirl...@sysdroid.com <mailto:gtirl...@sysdroid.com> ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Replacing a disk never completes
On 09/21/10 09:16 AM, Ben Miller wrote: On 09/20/10 10:45 AM, Giovanni Tirloni wrote: On Thu, Sep 16, 2010 at 9:36 AM, Ben Miller mailto:bmil...@mail.eecis.udel.edu>> wrote: I have an X4540 running b134 where I'm replacing 500GB disks with 2TB disks (Seagate Constellation) and the pool seems sick now. The pool has four raidz2 vdevs (8+2) where the first set of 10 disks were replaced a few months ago. I replaced two disks in the second set (c2t0d0, c3t0d0) a couple of weeks ago, but have been unable to get the third disk to finish replacing (c4t0d0). I have tried the resilver for c4t0d0 four times now and the pool also comes up with checksum errors and a permanent error (:<0x0>). The first resilver was from 'zpool replace', which came up with checksum errors. I cleared the errors which triggered the second resilver (same result). I then did a 'zpool scrub' which started the third resilver and also identified three permanent errors (the two additional were in files in snapshots which I then destroyed). I then did a 'zpool clear' and then another scrub which started the fourth resilver attempt. This last attempt identified another file with errors in a snapshot that I have now destroyed. Any ideas how to get this disk finished being replaced without rebuilding the pool and restoring from backup? The pool is working, but is reporting as degraded and with checksum errors. [...] Try to run a `zpool clear pool2` and see if clears the errors. If not, you may have to detach `c4t0d0s0/o`. I believe it's a bug that was fixed in recent builds. I had tried a clear a few times with no luck. I just did a detach and that did remove the old disk and has now triggered another resilver which hopefully works. I had tried a remove rather than a detach before, but that doesn't work on raidz2... thanks, Ben I made some progress. That resilver completed with 4 errors. I cleared those and still had the one error ":<0x0>" so I started a scrub. The scrub restarted the resilver on c4t0d0 again though! There currently are no errors anyway, but the resilver will be running for the next day+. Is this another bug or will doing a scrub eventually lead to a scrub of the pool instead of the resilver? Ben ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Disk keeps resilvering, was: Replacing a disk never completes
On 09/22/10 04:27 PM, Ben Miller wrote: On 09/21/10 09:16 AM, Ben Miller wrote: I had tried a clear a few times with no luck. I just did a detach and that did remove the old disk and has now triggered another resilver which hopefully works. I had tried a remove rather than a detach before, but that doesn't work on raidz2... thanks, Ben I made some progress. That resilver completed with 4 errors. I cleared those and still had the one error ":<0x0>" so I started a scrub. The scrub restarted the resilver on c4t0d0 again though! There currently are no errors anyway, but the resilver will be running for the next day+. Is this another bug or will doing a scrub eventually lead to a scrub of the pool instead of the resilver? Ben Well not much progress. The one permanent error ":<0x0>" came back. And the disk keeps wanting to resilver when trying to do a scrub. Now after the last resilver I have more checksum errors on the pool, but not on any disks: NAME STATE READ WRITE CKSUM pool2 ONLINE 0 037 ... raidz2-1ONLINE 0 074 All other checksum totals are 0. So three problems: 1. How to get the disk to stop resilvering? 2. How do you get checksum errors on the pool, but no disk is identified? If I clear them and let the resilver go again more checksum errors appear. So how to get rid of these errors? 3. How to get rid of the metadata:0x0 error? I'm currently destroying old snapshots (though that bug was fixed quite awhile ago and I'm running b134). I can try unmounting filesystems and remounting next (all are currently mounted). I can also schedule a reboot for next week if anyone things that would help. thanks, Ben ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Running on Dell hardware?
If you're still having issues go into the BIOS and disable C-States, if you haven't already. It is responsible for most of the problems with 11th Gen PowerEdge. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] How to delete hundreds of emtpy snapshots
zfs list is mighty slow on systems with a large number of objects, but there is no foreseeable plan that I'm aware of to solve that "problem". Never the less, you need to do a zfs list, therefore, do it once and work from that. zfs list > /tmp/zfs.out for i in `grep mydataset@ /tmp/zfs.out`; do zfs destroy $i; done As for 5 minute snapshots this is NOT a bad idea. It is, however, complex to manage. Thus, you need to employ tactics to make it more digestible. You need to ask yourself first why you want 5 min snaps. Is it replication? If so, create it, replicate it, destroy all but the last snapshot or even rotate them. Or, is it fallback in case you make a mistake? Then just keep around the last 6 snapshots or so. zfs rename & zfs destroy are your friends use them wisely. :) If you want to discuss exactly what your trying to facilitate I'm sure we can come up with some more concrete ideas to help you. benr. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ARCSTAT Kstat Definitions
Would someone "in the know" be willing to write up (preferably blog) definitive definitions/explanations of all the arcstats provided via kstat? I'm struggling with proper interpretation of certain values, namely "p", "memory_throttle_count", and the mru/mfu+ghost hit vs demand/prefetch hit counters. I think I've got it figured out, but I'd really like expert clarification before I start tweaking. Thanks. benr. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ARCSTAT Kstat Definitions
Thanks, not as much as I was hoping for but still extremely helpful. Can you, or others have a look at this: http://cuddletech.com/arc_summary.html This is a PERL script that uses kstats to drum up a report such as the following: System Memory: Physical RAM: 32759 MB Free Memory : 10230 MB LotsFree: 511 MB ARC Size: Current Size: 7989 MB (arcsize) Target Size (Adaptive): 8192 MB (c) Min Size (Hard Limit):1024 MB (zfs_arc_min) Max Size (Hard Limit):8192 MB (zfs_arc_max) ARC Size Breakdown: Most Recently Used Cache Size: 13%1087 MB (p) Most Frequently Used Cache Size:86%7104 MB (c-p) ARC Efficency: Cache Access Total: 3947194710 Cache Hit Ratio: 99% 3944674329 Cache Miss Ratio: 0% 2520381 Data Demand Efficiency:99% Data Prefetch Efficiency:69% CACHE HITS BY CACHE LIST: Anon:0%16730069 Most Frequently Used: 99%3915830091 (mfu) Most Recently Used: 0%10490502 (mru) Most Frequently Used Ghost: 0%439554 (mfu_ghost) Most Recently Used Ghost:0%1184113 (mru_ghost) CACHE HITS BY DATA TYPE: Demand Data:99%3914527790 Prefetch Data: 0%2447831 Demand Metadata: 0%10709326 Prefetch Metadata: 0%16989382 CACHE MISSES BY DATA TYPE: Demand Data:45%1144679 Prefetch Data: 42%1068975 Demand Metadata: 5%132649 Prefetch Metadata: 6%174078 - Feedback and input is welcome, in particular if I'm mischarrectorizing data. benr. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ARCSTAT Kstat Definitions
Its a starting point anyway. The key is to try and draw useful conclusions from the info to answer the torrent of "why is my ARC 30GB???" There are several things I'm unclear on whether or not I'm properly interpreting such as: * As you state, the anon pages. Even the comment in code is, to me anyway, a little vague. I include them because otherwise you look at the hit counters and wonder where a large chunk of them went. * Prefetch... I want to use the Prefetch Data hit ratio as a judgment call on the efficiency of prefetch. If the value is very low it might be best to turn it off. but I'd like to hear that from someone else before I go saying that. In high latency environments, such as ZFS on iSCSI, prefetch can either significantly help or hurt, determining which is difficult without some type of metric as as above. * There are several instances (based on dtracing) in which the ARC is bypassed... for ZIL I understand, in some other cases I need to spend more time analyzing the DMU (dbuf_*) for why. * In answering the "Is having a 30GB ARC good?" question, I want to say that if MFU is >60% of ARC, and if the hits are mostly MFU that you are deriving significant benefit from your large ARC but on a system with a 2GB ARC or a 30GB ARC the overall hit ratio tends to be 99%. Which is nuts, and tends to reinforce a misinterpretation of anon hits. The only way I'm seeing to _really_ understand ARC's efficiency is to look at the overall number of reads and then how many are intercepted by ARC and how many actually made it to disk... and why (prefetch or demand). This is tricky to implement via kstats because you have to pick out and monitor the zpool disks themselves. I've spent a lot of time in this code (arc.c) and still have a lot of questions. I really wish there was an "Advanced ZFS Internals" talk coming up; I simply can't keep spending so much time on this. Feedback from PAE or other tuning experts is welcome and appreciated. :) benr. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ARCSTAT Kstat Definitions
New version is available (v0.2) : * Fixes divide by zero, * includes tuning from /etc/system in output * if prefetch is disabled I explicitly say so. * Accounts for jacked anon count. Still need improvement here. * Added friendly explanations for MRU/MFU & Ghost lists counts. Page and examples are updated: cuddletech.com/arc_summary.pl Still needs work, but hopefully interest in this will stimulate some improved understanding of ARC internals. benr. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Couple of ZFS panics...
I've got a Intel DP35DP Motherboard, Q6600 proc (Intel 2.4G, 4 core), 4GB of ram and a copule of Sata disks, running ICH9. S10U5, patched about a week ago or so... I have a zpool on a single slice (haven't added a mirror yet, was getting to that) and have started to suffer regular hard resets and have gotten a few panics. The system is an nfs server for a couple of systems (not much write) and one writer (I do my svn updates over NFS cause my ath0 board refuses to work in 64-bit on S10U5) I also do local builds on the same server. Ideas? The first looks like: panic[cpu0]/thread=9bcf0460: BAD TRAP: type=e (#pf Page fault) rp=fe80001739a0 addr=c064dba0 cmake: #pf Page fault Bad kernel fault at addr=0xc064dba0 pid=6797, pc=0xf0a6350a, sp=0xfe8000173a90, eflags=0x10207 cr0: 80050033 cr4: 6f8 cr2: c064dba0 cr3: 12bf9b000 cr8: c rdi: 6c60 rsi:0 rdx:0 rcx:0 r8: 8b21017f r9: ae3a79c0 rax:0 rbx: c0611f40 rbp: fe8000173ac0 r10:0 r11:0 r12: ae4687d0 r13: d8c200 r14:2 r15: 826c0480 fsb: 8000 gsb: fbc24ec0 ds: 43 es: 43 fs:0 gs: 1c3 trp:e err:0 rip: f0a6350a cs: 28 rfl:10207 rsp: fe8000173a90 ss: 30 fe80001738b0 unix:real_mode_end+71e1 () fe8000173990 unix:trap+5e6 () fe80001739a0 unix:_cmntrap+140 () fe8000173ac0 zfs:zio_buf_alloc+a () fe8000173af0 zfs:arc_buf_alloc+9f () fe8000173b70 zfs:arc_read+ee () fe8000173bf0 zfs:dbuf_read_impl+1a0 () fe8000173c30 zfs:zfsctl_ops_root+304172dd () fe8000173c60 zfs:dmu_tx_check_ioerr+6e () fe8000173cc0 zfs:dmu_tx_count_write+73 () fe8000173cf0 zfs:dmu_tx_hold_write+4a () fe8000173db0 zfs:zfs_write+1bb () fe8000173e00 genunix:fop_write+31 () fe8000173eb0 genunix:write+287 () fe8000173ec0 genunix:write32+e () fe8000173f10 unix:brand_sys_sysenter+1f2 () syncing file systems... 3130 15 done dumping to /dev/dsk/c0t0d0s1, offset 860356608, content: kernel NOTICE: ahci_tran_reset_dport: port 0 reset port The second liek this: panic[cpu2]/thread=9b425f20: BAD TRAP: type=e (#pf Page fault) rp=fe80018cdf40 addr=c064dba0 nfsd: #pf Page fault Bad kernel fault at addr=0xc064dba0 pid=665, pc=0xf0a6350a, sp=0xfe80018ce030, eflags=0x10207 cr0: 8005003b cr4: 6f8 cr2: c064dba0 cr3: 12a9df000 cr8: c rdi: 6c60 rsi:0 rdx:0 rcx:0 r8: 8b21017f r9:f rax:0 rbx: c0611f40 rbp: fe80018ce060 r10:0 r11:0 r12: fe81c20ecf00 r13: d8c200 r14:2 r15: 826c2240 fsb: 8000 gsb: 81a6c800 ds: 43 es: 43 fs:0 gs: 1c3 trp:e err:0 rip: f0a6350a cs: 28 rfl:10207 rsp: fe80018ce030 ss: 30 fe80018cde50 unix:real_mode_end+71e1 () fe80018cdf30 unix:trap+5e6 () fe80018cdf40 unix:_cmntrap+140 () fe80018ce060 zfs:zio_buf_alloc+a () fe80018ce090 zfs:arc_buf_alloc+9f () fe80018ce110 zfs:arc_read+ee () fe80018ce190 zfs:dbuf_read_impl+1a0 () fe80018ce1d0 zfs:zfsctl_ops_root+304172dd () fe80018ce200 zfs:dmu_tx_check_ioerr+6e () fe80018ce260 zfs:dmu_tx_count_write+73 () fe80018ce290 zfs:dmu_tx_hold_write+4a () fe80018ce350 zfs:zfs_write+1bb () fe80018ce3a0 genunix:fop_write+31 () fe80018ce410 nfssrv:do_io+b5 () fe80018ce610 nfssrv:rfs4_op_write+40e () fe80018ce770 nfssrv:rfs4_compound+1b3 () fe80018ce800 nfssrv:rfs4_dispatch+234 () fe80018ceb10 nfssrv:common_dispatch+88a () fe80018ceb20 nfssrv:nfs4_drc+3051ccc1 () fe80018cebf0 rpcmod:svc_getreq+209 () fe80018cec40 rpcmod:svc_run+124 () fe80018cec70 rpcmod:svc_do_run+88 () fe80018ceec0 nfs:nfssys+208 () fe80018cef10 unix:brand_sys_sysenter+1f2 () syncing file systems... done dumping to /dev/dsk/c0t0d0s1, offset 860356608, content: kernel NOTICE: ahci_tran_reset_dport: port 0 reset port This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Lost Disk Space
I've been struggling to fully understand why disk space seems to vanish. I've dug through bits of code and reviewed all the mails on the subject that I can find, but I still don't have a proper understanding of whats going on. I did a test with a local zpool on snv_97... zfs list, zpool list, and zdb all seem to disagree on how much space is available. In this case its only a discrepancy of about 20G or so, but I've got Thumpers that have a discrepancy of over 6TB! Can someone give a really detailed explanation about whats going on? block traversal size 670225837056 != alloc 720394438144 (leaked 50168601088) bp count:15182232 bp logical:672332631040 avg: 44284 bp physical: 669020836352 avg: 44066compression: 1.00 bp allocated: 670225837056 avg: 44145compression: 1.00 SPA allocated: 720394438144 used: 96.40% Blocks LSIZE PSIZE ASIZE avgcomp %Total Type 12 120K 26.5K 79.5K 6.62K4.53 0.00 deferred free 1512 512 1.50K 1.50K1.00 0.00 object directory 3 1.50K 1.50K 4.50K 1.50K1.00 0.00 object array 116K 1.50K 4.50K 4.50K 10.67 0.00 packed nvlist - - - - - -- packed nvlist size 72 8.45M889K 2.60M 37.0K9.74 0.00 bplist - - - - - -- bplist header - - - - - -- SPA space map header 974 4.48M 2.65M 7.94M 8.34K1.70 0.00 SPA space map - - - - - -- ZIL intent log 96.7K 1.51G389M777M 8.04K3.98 0.12 DMU dnode 17 17.0K 8.50K 17.5K 1.03K2.00 0.00 DMU objset - - - - - -- DSL directory 13 6.50K 6.50K 19.5K 1.50K1.00 0.00 DSL directory child map 12 6.00K 6.00K 18.0K 1.50K1.00 0.00 DSL dataset snap map 14 38.0K 10.0K 30.0K 2.14K3.80 0.00 DSL props - - - - - -- DSL dataset - - - - - -- ZFS znode 2 1K 1K 2K 1K1.00 0.00 ZFS V0 ACL 5.81M 558G557G557G 95.8K1.0089.27 ZFS plain file 382K 301M200M401M 1.05K1.50 0.06 ZFS directory 9 4.50K 4.50K 9.00K 1K1.00 0.00 ZFS master node 12 482K 20.0K 40.0K 3.33K 24.10 0.00 ZFS delete queue 8.20M 66.1G 65.4G 65.8G 8.03K1.0110.54 zvol object 1512 512 1K 1K1.00 0.00 zvol prop - - - - - -- other uint8[] - - - - - -- other uint64[] - - - - - -- other ZAP - - - - - -- persistent error log 1 128K 10.5K 31.5K 31.5K 12.19 0.00 SPA history - - - - - -- SPA history offsets - - - - - -- Pool properties - - - - - -- DSL permissions - - - - - -- ZFS ACL - - - - - -- ZFS SYSACL - - - - - -- FUID table - - - - - -- FUID table size 5 3.00K 2.50K 7.50K 1.50K1.20 0.00 DSL dataset next clones - - - - - -- scrub work queue 14.5M 626G623G624G 43.1K1.00 100.00 Total real21m16.862s user0m36.984s sys 0m5.757s === Looking at the data: [EMAIL PROTECTED] ~$ zfs list backup && zpool list backup NAME USED AVAIL REFER MOUNTPOINT backup 685G 237K27K /backup NAME SIZE USED AVAILCAP HEALTH ALTROOT backup 696G 671G 25.1G96% ONLINE - So zdb says 626GB is used, zfs list says 685GB is used, and zpool list says 671GB is used. The pool was filled to 100% capacity via dd, this is confirmed, I can't write data, but yet zpool list says its only 96%. benr. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Lost Disk Space
No takers? :) benr. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] zdb to dump data
Is there some hidden way to coax zdb into not just displaying data based on a given DVA but rather to dump it in raw usable form? I've got a pool with large amounts of corruption. Several directories are toast and I get "I/O Error" when trying to enter or read the directory... however I can read the directory and files using ZDB, if I could just dump it in a raw format I could do recovery that way. To be clear, I've already recovered from the situation, this is purely an academic "can I do it" exercise for the sake of learning. If ZDB can't do it, I'd assume I'd have to write some code to read based on DVA. Maybe I could write a little tool for it. benr. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool status -x strangeness
This post from close to a year ago never received a response. We just had this same thing happen to another server that is running Solaris 10 U6. One of the disks was marked as removed and the pool degraded, but 'zpool status -x' says all pools are healthy. After doing an 'zpool online' on the disk it resilvered in fine. Any ideas why 'zpool status -x' reports all healthy while 'zpool status' shows a pool in degraded mode? thanks, Ben > We run a cron job that does a 'zpool status -x' to > check for any degraded pools. We just happened to > find a pool degraded this morning by running 'zpool > status' by hand and were surprised that it was > degraded as we didn't get a notice from the cron > job. > > # uname -srvp > SunOS 5.11 snv_78 i386 > > # zpool status -x > all pools are healthy > > # zpool status pool1 > pool: pool1 > tate: DEGRADED > scrub: none requested > onfig: > > NAME STATE READ WRITE CKSUM > pool1DEGRADED 0 0 0 > raidz1 DEGRADED 0 0 0 > c1t8d0 REMOVED 0 0 0 > c1t9d0 ONLINE 0 0 0 > c1t10d0 ONLINE 0 0 0 > c1t11d0 ONLINE 0 0 0 > No known data errors > > I'm going to look into it now why the disk is listed > as removed. > > Does this look like a bug with 'zpool status -x'? > > Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool status -x strangeness
I just put in a (low priority) bug report on this. Ben > This post from close to a year ago never received a > response. We just had this same thing happen to > another server that is running Solaris 10 U6. One of > the disks was marked as removed and the pool > degraded, but 'zpool status -x' says all pools are > healthy. After doing an 'zpool online' on the disk > it resilvered in fine. Any ideas why 'zpool status > -x' reports all healthy while 'zpool status' shows a > pool in degraded mode? > > thanks, > Ben > > > We run a cron job that does a 'zpool status -x' to > > check for any degraded pools. We just happened to > > find a pool degraded this morning by running > 'zpool > > status' by hand and were surprised that it was > > degraded as we didn't get a notice from the cron > > job. > > > > # uname -srvp > > SunOS 5.11 snv_78 i386 > > > > # zpool status -x > > all pools are healthy > > > > # zpool status pool1 > > pool: pool1 > > tate: DEGRADED > > scrub: none requested > > onfig: > > > > NAME STATE READ WRITE CKSUM > > pool1DEGRADED 0 0 0 > > raidz1 DEGRADED 0 0 0 > > c1t8d0 REMOVED 0 0 0 > > c1t9d0 ONLINE 0 0 0 > > c1t10d0 ONLINE 0 0 0 > > c1t11d0 ONLINE 0 0 0 > > No known data errors > > > > I'm going to look into it now why the disk is > listed > > as removed. > > > > Does this look like a bug with 'zpool status -x'? > > > > Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool status -x strangeness
Bug ID is 6793967. This problem just happened again. % zpool status pool1 pool: pool1 state: DEGRADED scrub: resilver completed after 0h48m with 0 errors on Mon Jan 5 12:30:52 2009 config: NAME STATE READ WRITE CKSUM pool1 DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 c4t8d0s0 ONLINE 0 0 0 c4t9d0s0 ONLINE 0 0 0 c4t10d0s0 ONLINE 0 0 0 c4t11d0s0 ONLINE 0 0 0 c4t12d0s0 REMOVED 0 0 0 c4t13d0s0 ONLINE 0 0 0 errors: No known data errors % zpool status -x all pools are healthy % # zpool online pool1 c4t12d0s0 % zpool status -x pool: pool1 state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress for 0h0m, 0.12% done, 2h38m to go config: NAME STATE READ WRITE CKSUM pool1 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c4t8d0s0 ONLINE 0 0 0 c4t9d0s0 ONLINE 0 0 0 c4t10d0s0 ONLINE 0 0 0 c4t11d0s0 ONLINE 0 0 0 c4t12d0s0 ONLINE 0 0 0 c4t13d0s0 ONLINE 0 0 0 errors: No known data errors % Ben > I just put in a (low priority) bug report on this. > > Ben > > > This post from close to a year ago never received > a > > response. We just had this same thing happen to > > another server that is running Solaris 10 U6. One > of > > the disks was marked as removed and the pool > > degraded, but 'zpool status -x' says all pools are > > healthy. After doing an 'zpool online' on the > disk > > it resilvered in fine. Any ideas why 'zpool > status > > -x' reports all healthy while 'zpool status' shows > a > > pool in degraded mode? > > > > thanks, > > Ben > > > > > We run a cron job that does a 'zpool status -x' > to > > > check for any degraded pools. We just happened > to > > > find a pool degraded this morning by running > > 'zpool > > > status' by hand and were surprised that it was > > > degraded as we didn't get a notice from the cron > > > job. > > > > > > # uname -srvp > > > SunOS 5.11 snv_78 i386 > > > > > > # zpool status -x > > > all pools are healthy > > > > > > # zpool status pool1 > > > pool: pool1 > > > tate: DEGRADED > > > scrub: none requested > > > onfig: > > > > > > NAME STATE READ WRITE CKSUM > > > pool1DEGRADED 0 0 0 > > > raidz1 DEGRADED 0 0 0 > > > c1t8d0 REMOVED 0 0 0 > > > c1t9d0 ONLINE 0 0 0 > > > c1t10d0 ONLINE 0 0 0 > > > c1t11d0 ONLINE 0 0 0 > > > No known data errors > > > > > > I'm going to look into it now why the disk is > > listed > > > as removed. > > > > > > Does this look like a bug with 'zpool status > -x'? > > > > > > Ben -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool status -x strangeness
The pools are upgraded to version 10. Also, this is on Solaris 10u6. # zpool upgrade This system is currently running ZFS pool version 10. All pools are formatted using this version. Ben > What's the output of 'zfs upgrade' and 'zpool > upgrade'? (I'm just > curious - I had a similar situation which seems to be > resolved now > that I've gone to Solaris 10u6 or OpenSolaris > 2008.11). > > > > On Wed, Jan 21, 2009 at 2:11 PM, Ben Miller > wrote: > > Bug ID is 6793967. > > > > This problem just happened again. > > % zpool status pool1 > > pool: pool1 > > state: DEGRADED > > scrub: resilver completed after 0h48m with 0 > errors on Mon Jan 5 12:30:52 2009 > > config: > > > >NAME STATE READ WRITE CKSUM > >pool1 DEGRADED 0 0 0 > > raidz2 DEGRADED 0 0 0 > >c4t8d0s0 ONLINE 0 0 0 > >c4t9d0s0 ONLINE 0 0 0 > >c4t10d0s0 ONLINE 0 0 0 > >c4t11d0s0 ONLINE 0 0 0 > >c4t12d0s0 REMOVED 0 0 0 > >c4t13d0s0 ONLINE 0 0 0 > > > > errors: No known data errors > > > > % zpool status -x > > all pools are healthy > > % > > # zpool online pool1 c4t12d0s0 > > % zpool status -x > > pool: pool1 > > state: ONLINE > > status: One or more devices is currently being > resilvered. The pool will > >continue to function, possibly in a degraded > state. > > action: Wait for the resilver to complete. > > scrub: resilver in progress for 0h0m, 0.12% done, > 2h38m to go > > config: > > > >NAME STATE READ WRITE CKSUM > >pool1 ONLINE 0 0 0 > > raidz2 ONLINE 0 0 0 > >c4t8d0s0 ONLINE 0 0 0 > > c4t9d0s0 ONLINE 0 0 0 > >c4t10d0s0 ONLINE 0 0 0 > >c4t11d0s0 ONLINE 0 0 0 > >c4t12d0s0 ONLINE 0 0 0 > >c4t13d0s0 ONLINE 0 0 0 > > > > errors: No known data errors > > % > > > > Ben > > -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool status -x strangeness
We haven't done 'zfs upgrade ...' any. I'll give that a try the next time the system can be taken down. Ben > A little gotcha that I found in my 10u6 update > process was that 'zpool > upgrade [poolname]' is not the same as 'zfs upgrade > [poolname]/[filesystem(s)]' > > What does 'zfs upgrade' say? I'm not saying this is > the source of > your problem, but it's a detail that seemed to affect > stability for > me. > > > On Thu, Jan 22, 2009 at 7:25 AM, Ben Miller > > The pools are upgraded to version 10. Also, this > is on Solaris 10u6. > > -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool status -x strangeness
I forgot the pool that's having problems was recreated recently so it's already at zfs version 3. I just did a 'zfs upgrade -a' for another pool, but some of those filesystems failed since they are busy and couldn't be unmounted. # zfs upgrade -a cannot unmount '/var/mysql': Device busy cannot unmount '/var/postfix': Device busy 6 filesystems upgraded 821 filesystems already at this version Ben > You can upgrade live. 'zfs upgrade' with no > arguments shows you the > zfs version status of filesystems present without > upgrading. > > > > On Jan 24, 2009, at 10:19 AM, Ben Miller > wrote: > > > We haven't done 'zfs upgrade ...' any. I'll give > that a try the > > next time the system can be taken down. > > > > Ben > > > >> A little gotcha that I found in my 10u6 update > >> process was that 'zpool > >> upgrade [poolname]' is not the same as 'zfs > upgrade > >> [poolname]/[filesystem(s)]' > >> > >> What does 'zfs upgrade' say? I'm not saying this > is > >> the source of > >> your problem, but it's a detail that seemed to > affect > >> stability for > >> me. > >> > >> -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool status -x strangeness
# zpool status -xv all pools are healthy Ben > What does 'zpool status -xv' show? > > On Tue, Jan 27, 2009 at 8:01 AM, Ben Miller > wrote: > > I forgot the pool that's having problems was > recreated recently so it's already at zfs version 3. > I just did a 'zfs upgrade -a' for another pool, but > some of those filesystems failed since they are busy > and couldn't be unmounted. > > > # zfs upgrade -a > > cannot unmount '/var/mysql': Device busy > > cannot unmount '/var/postfix': Device busy > > > > 6 filesystems upgraded > > 821 filesystems already at this version > > > > Ben > > -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [Fwd: Re: [perf-discuss] ZFS performance issue - READ is slow as hell...]
Ya, I agree that we need some additional data and testing. The iostat data in itself doesn't suggest to me that the process (dd) is slow but rather that most of the data is being retrieved elsewhere (ARC). An fsstat would be useful to correlate with the iostat data. One thing that also comes to mind with streaming write performance is the effects of the write throttle... curious if he'd have gotten more on the write side with that disabled. All these things don't strike me particularly as bugs (although there is always improvement) but rather that ZFS is designed for real world environments, not antiquated benchmarks. benr. Jim Mauro wrote: > > Posting this back to zfs-discuss. > > Roland's test case (below) is a single threaded sequential write > followed by a single threaded sequential read. His bandwidth > goes from horrible (~2MB/sec) to expected (~30MB/sec) > when prefetch is disabled. This is with relatively recent nv bits > (nv110). > > Roland - I'm wondering if you were tripping over > CR6732803 ZFS prefetch creates performance issues for streaming > workloads. > It seems possible, but that CR is specific about multiple, concurrent > IO streams, > and your test case was only one. > > I think it's more likely you were tripping over > CR6412053 zfetch needs a whole lotta love. > > For both CR's the workaround is disabling prefetch > (echo "zfs_prefetch_disable/W 1" | mdb -kw) > > Any other theories on this test case? > > Thanks, > /jim > > > Original Message > Subject: Re: [perf-discuss] ZFS performance issue - READ is slow > as hell... > Date: Tue, 31 Mar 2009 02:33:00 -0700 (PDT) > From: roland > To: perf-disc...@opensolaris.org > > > > Hello Jim, > i double checked again - but it`s like i told: > > echo zfs_prefetch_disable/W0t1 | mdb -kw > fixes my problem. > > i did a reboot and only set this single param - which immediately > makes the read troughput go up from ~2 MB/s to ~30 MB/s > >> I don't understand why disabling ZFS prefetch solved this >> problem. The test case was a single threaded sequential write, followed >> by a single threaded sequential read. > > i did not even do a single write - after reboot i just did > dd if=/zfs/TESTFILE of=/dev/null > > Solaris Express Community Edition snv_110 X86 > FSC RX300 S2 > 4GB RAM > LSI Logic MegaRaid 320 Onboard SCSI Raid Controller > 1x Raid1 LUN > 1x Raid5 LUN (3 Disks) > (both LUN`s show same behaviour) > > > before: > extended device statistics > r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 21.30.1 2717.60.1 0.7 0.0 31.81.7 2 4 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 > 16.00.0 2048.40.0 34.9 0.1 2181.84.8 100 3 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 > 28.00.0 3579.20.0 34.8 0.1 1246.24.9 100 5 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 > 45.00.0 5760.40.0 34.8 0.2 772.74.5 100 7 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 > 19.00.0 2431.90.0 34.9 0.1 1837.34.4 100 3 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 > 58.00.0 7421.10.0 34.6 0.3 597.45.8 100 12 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 >0.00.00.00.0 35.0 0.00.00.0 100 0 c0t1d0 > > > after: > extended device statistics > r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 218.00.0 27842.30.0 0.0 0.40.11.8 1 40 c0t1d0 > 241.00.0 30848.00.0 0.0 0.40.01.6 0 38 c0t1d0 > 237.00.0 30340.10.0 0.0 0.40.01.6 0 38 c0t1d0 > 230.00.0 29434.70.0 0.0 0.40.01.8 0 40 c0t1d0 > 238.10.0 30471.30.0 0.0 0.40.01.5 0 37 c0t1d0 > 234.90.0
Re: [zfs-discuss] Re: Thumper Origins Q
On Jan 24, 2007, at 12:37 PM, Shannon Roddy wrote: I went with a third party FC/SATA unit which has been flawless as a direct attach for my ZFS JBOD system. Paid about $0.70/GB. What did you use, if you don't mind my asking? -- Ben PGP.sig Description: This is a digitally signed message part ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs / nfs issue (not performance :-) with courier-imap
Robert Milkowski wrote: CLSNL> but if I click, say E, it has F's contents, F has Gs contents, and no CLSNL> mail has D's contents that I can see. But the list in the mail CLSNL> client list view is correct. I don't belive it's a problem with nfs/zfs server. Please try with simple dtrace script to see (or even truss) what files your imapd actually opens when you click E - I don't belive it opens E and you get F contents, I would bet it opens F. I completely agree with Robert. I'd personally suggest 'truss' to start because its trivial to use, then start using DTrace to further hone down the problem. In the case of Courier-IMAP the best way to go about it would be to truss the parent (courierlogger, which calls courierlogin and ultimately imapd) using 'truss -f -p '. Then open the mailbox and watch those stat's and open's closely. I'll be very interested in your findings. We use Courier on NFS/ZFS heavily and I'm thankful to report having no such problems. benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Read Only Zpool: ZFS and Replication
I've been playing with replication of a ZFS Zpool using the recently released AVS. I'm pleased with things, but just replicating the data is only part of the problem. The big question is: can I have a zpool open in 2 places? What I really want is a Zpool on node1 open and writable (production storage) and a replicated to node2 where its open for read-only access (standby storage). This is an old problem. I'm not sure its remotely possible. Its bad enough with UFS, but ZFS maintains a hell of a lot more meta-data. How is node2 supposed to know that a snapshot has been created for instance. With UFS you can at least get by some of these problems using directio, but thats not an option with a zpool. I know this is a fairly remedial issue to bring up... but if I think about what I want Thumper-to-Thumper replication to look like, I want 2 usable storage systems. As I see it now the secondary storage (node2) is useless untill you break replication and import the pool, do your thing, and then re-sync storage to re-enable replication. Am I missing something? I'm hoping there is an option I'm not aware of. benr. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] snapdir visable recursively throughout a dataset
Is there an existing RFE for, what I'll wrongly call, "recursively visable snapshots"? That is, .zfs in directories other than the dataset root. Frankly, I don't need it available in all directories, although it'd be nice, but I do have a need for making it visiable 1 dir down from the dataset root. The problem is that while ZFS and Zones work smoothly together for moving, cloning, sizing, etc, you can't view .zfs/ from within the zone because the zone root is one dir down: /zones <-- Dataset /zones/myzone01 <-- Dataset, .zfs is located here. /zones/myzone01/root <-- Directory, want .zfs Here! The ultimate idea is to make ZFS snapdirs accessable from within the zone. benr. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Read Only Zpool: ZFS and Replication
Jim Dunham wrote: Robert, Hello Ben, Monday, February 5, 2007, 9:17:01 AM, you wrote: BR> I've been playing with replication of a ZFS Zpool using the BR> recently released AVS. I'm pleased with things, but just BR> replicating the data is only part of the problem. The big BR> question is: can I have a zpool open in 2 places? BR> What I really want is a Zpool on node1 open and writable BR> (production storage) and a replicated to node2 where its open for BR> read-only access (standby storage). BR> This is an old problem. I'm not sure its remotely possible. Its BR> bad enough with UFS, but ZFS maintains a hell of a lot more BR> meta-data. How is node2 supposed to know that a snapshot has been BR> created for instance. With UFS you can at least get by some of BR> these problems using directio, but thats not an option with a zpool. BR> I know this is a fairly remedial issue to bring up... but if I BR> think about what I want Thumper-to-Thumper replication to look BR> like, I want 2 usable storage systems. As I see it now the BR> secondary storage (node2) is useless untill you break replication BR> and import the pool, do your thing, and then re-sync storage to re-enable replication. BR> Am I missing something? I'm hoping there is an option I'm not aware of. You can't mount rw on one node and ro on another (not to mention that zfs doesn't offer you to import RO pools right now). You can mount the same file system like UFS in RO on both nodes but not ZFS (no ro import). One can not just mount a filesystem in RO mode if SNDR or any other host-based or controller-based replication is underneath. For all filesystems that I know of, expect of course shared-reader QFS, this will fail given time. Even if one has the means to mount a filesystem with DIRECTIO (no-caching), READ-ONLY (no-writes), it does not prevent a filesystem from looking at the contents of block "A" and then acting on block "B". The reason being is that during replication at time T1 both blocks "A" & "B" could be written and be consistent with each other. Next the file system reads block "A". Now replication at time T2 updates blocks "A" & "B", also consistent with each other. Next the file system reads block "B" and panics due to an inconsistency only it sees between old "A" and new "B". I know this for a fact, since a forced "zpool import -f ", is a common instance of this exact failure, due most likely checksum failures between metadata blocks "A" & "B". Ya, that bit me last night. 'zpool import' shows the pool fine, but when you force the import you panic: Feb 5 07:14:10 uma ^Mpanic[cpu0]/thread=fe8001072c80: Feb 5 07:14:10 uma genunix: [ID 809409 kern.notice] ZFS: I/O failure (write on off 0: zio fe80c54ed380 [L0 unallocated] 400L/200P DVA[0]=<0:36000:200> DVA[1]=<0:9c0003800:200> DVA[2]=<0:20004e00:200> fletcher4 lzjb LE contiguous birth=57416 fill=0 cksum=de2e56ffd:5591b77b74b:1101a91d58dfc:252efdf22532d0): error 5 Feb 5 07:14:11 uma unix: [ID 10 kern.notice] Feb 5 07:14:11 uma genunix: [ID 655072 kern.notice] fe8001072a40 zfs:zio_done+140 () Feb 5 07:14:11 uma genunix: [ID 655072 kern.notice] fe8001072a60 zfs:zio_next_stage+68 () Feb 5 07:14:11 uma genunix: [ID 655072 kern.notice] fe8001072ab0 zfs:zio_wait_for_children+5d () Feb 5 07:14:11 uma genunix: [ID 655072 kern.notice] fe8001072ad0 zfs:zio_wait_children_done+20 () Feb 5 07:14:11 uma genunix: [ID 655072 kern.notice] fe8001072af0 zfs:zio_next_stage+68 () Feb 5 07:14:11 uma genunix: [ID 655072 kern.notice] fe8001072b40 zfs:zio_vdev_io_assess+129 () Feb 5 07:14:11 uma genunix: [ID 655072 kern.notice] fe8001072b60 zfs:zio_next_stage+68 () Feb 5 07:14:11 uma genunix: [ID 655072 kern.notice] fe8001072bb0 zfs:vdev_mirror_io_done+2af () Feb 5 07:14:11 uma genunix: [ID 655072 kern.notice] fe8001072bd0 zfs:zio_vdev_io_done+26 () Feb 5 07:14:11 uma genunix: [ID 655072 kern.notice] fe8001072c60 genunix:taskq_thread+1a7 () Feb 5 07:14:11 uma genunix: [ID 655072 kern.notice] fe8001072c70 unix:thread_start+8 () Feb 5 07:14:11 uma unix: [ID 10 kern.notice] So without using II, whats the best method of bring up the secondary storage? Is just dropping the primary into logging acceptable? benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] snapdir visable recursively throughout a dataset
Robert Milkowski wrote: I haven't tried it but what if you mounted ro via loopback into a zone /zones/myzone01/root/.zfs is loop mounted in RO to /zones/myzone01/.zfs That is so wrong. ;) Besides just being evil, I doubt it'd work. And if it does, it probly shouldn't. I think I'm the only one that gets a rash when using LOFI. benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] snapdir visable recursively throughout a dataset
Darren J Moffat wrote: Ben Rockwood wrote: Robert Milkowski wrote: I haven't tried it but what if you mounted ro via loopback into a zone /zones/myzone01/root/.zfs is loop mounted in RO to /zones/myzone01/.zfs That is so wrong. ;) Besides just being evil, I doubt it'd work. And if it does, it probly shouldn't. I think I'm the only one that gets a rash when using LOFI. lofi or lofs ? lofi - Loopback file driver Makes a block device from a file lofs - loopback virtual file system Makes a file system from a file system Yes, I know. I was referring more so to loopback happy people in general. :) benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] [OT] Multipathing on Mac OS X
This is pretty OT, but a bit ago there was some discussion of Mac OS X's multipathing support (or its lack thereof). According to this technote, multipathing support has been included Mac OS X since 10.3.5, but there are some particular requirements on the target devices & HBAs. http://developer.apple.com/technotes/tn2007/tn2173.html -- Ben PGP.sig Description: This is a digitally signed message part ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS disables nfs/server on a host
I have an Ultra 10 client running Sol10 U3 that has a zfs pool set up on the
extra space of the internal ide disk. There's just the one fs and it is shared
with the sharenfs property. When this system reboots nfs/server ends up
getting disabled and this is the error from the SMF logs:
[ Apr 16 08:41:22 Executing start method ("/lib/svc/method/nfs-server start") ]
[ Apr 16 08:41:24 Method "start" exited with status 0 ]
[ Apr 18 10:59:23 Executing start method ("/lib/svc/method/nfs-server start") ]
Assertion failed: pclose(fp) == 0, file ../common/libzfs_mount.c, line 380,
function zfs_share
If I re-enable nfs/server after the system is up it's fine. The system was
recently upgraded to use zfs and this has happened on the last two reboots. We
have lots of other systems that share nfs through zfs fine and I didn't see a
similar problem on the list. Any ideas?
Ben
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: ZFS disables nfs/server on a host
It does seem like an ordering problem, but nfs/server should be starting up late enough with SMF dependencies. I need to see if I can duplicate the problem on a test system... This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] XServe Raid & Complex Storage Considerations
On Apr 25, 2007, at 10:54 AM, Andy Lubel wrote: The Xraid is a very well thought of storage device with a heck of a price point. Attached is an image of the "Settings"/"Performance" Screen where you see "Allow Host Cache Flushing". I think when you use ZFS, it would be best to uncheck that box. Yes, that's the setting you're looking for. The Xserve RAID works great with ZFS IME. -- Ben PGP.sig Description: This is a digitally signed message part ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: ZFS disables nfs/server on a host
I just rebooted this host this morning and the same thing happened again. I
have the core file from zfs.
[ Apr 26 07:47:01 Executing start method ("/lib/svc/method/nfs-server start") ]
Assertion failed: pclose(fp) == 0, file ../common/libzfs_mount.c, line 380, func
tion zfs_share
Abort - core dumped
Why would nfs/server be disabled instead of going into maintenance with this
error?
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: ZFS disables nfs/server on a host
I was able to duplicate this problem on a test Ultra 10. I put in a workaround by adding a service that depends on /milestone/multi-user-server which does a 'zfs share -a'. It's strange this hasn't happened on other systems, but maybe it's related to slower systems... Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Re: ZFS disables nfs/server on a host
I just threw in a truss in the SMF script and rebooted the test system and it failed again. The truss output is at http://www.eecis.udel.edu/~bmiller/zfs.truss-Apr27-2007 thanks, Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Remove files when at quota limit
We have around 1000 users all with quotas set on their ZFS filesystems on Solaris 10 U3. We take snapshots daily and rotate out the week old ones. The situation is that some users ignore the advice of keeping space used below 80% and keep creating large temporary files. They then try to remove files when the space used is 100% and get over quota messages. We then need to remove some or all of their snapshots to free space. Is there anything being worked on to keep some space reserved so files can be removed when at the quota limit or some other solution? What are other people doing is this situation? We have also set up alternate filesystems for users with transient data that we do not take snapshots on, but we still have this problem on home directories. thanks, Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Remove files when at quota limit
Has anyone else run into this situation? Does anyone have any solutions other than removing snapshots or increasing the quota? I'd like to put in an RFE to reserve some space so files can be removed when users are at their quota. Any thoughts from the ZFS team? Ben > We have around 1000 users all with quotas set on > their ZFS filesystems on Solaris 10 U3. We take > snapshots daily and rotate out the week old ones. > The situation is that some users ignore the advice > of keeping space used below 80% and keep creating > large temporary files. They then try to remove > files when the space used is 100% and get over quota > messages. We then need to remove some or all of > their snapshots to free space. Is there anything > being worked on to keep some space reserved so files > can be removed when at the quota limit or some other > solution? What are other people doing is this > situation? We have also set up alternate > filesystems for users with transient data that we do > not take snapshots on, but we still have this > problem on home directories. > > thanks, > Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Making 'zfs destroy' safer
Peter Schuller wrote: Hello, with the advent of clones and snapshots, one will of course start creating them. Which also means destroying them. Am I the only one who is *extremely* nervous about doing "zfs destroy some/[EMAIL PROTECTED]"? This goes bot manually and automatically in a script. I am very paranoid about this; especially because the @ sign might conceivably be incorrectly interpreted by some layer of scripting, being a non-alphanumeric character and highly atypical for filenames/paths. What about having dedicated commands "destroysnapshot", "destroyclone", or "remove" (less dangerous variant of "destroy") that will never do anything but remove snapshots or clones? Alternatively having something along the lines of "zfs destroy --nofs" or "zfs destroy --safe". I realize this is borderline being in the same territory as special casing "rm -rf /" and similar, which is generally not considered a good idea. But somehow the snapshot situation feels a lot more risky. This isn't the first time this subject has come up. You are definitely NOT alone. The problem is only compounded when doing recursive actions. The general request has been for a confirmation "Are you sure?" which could be over-ridden with a -f. The general response is "if run from a script everyone will use -f and defeat the purpose". The suggestions you've come up with above are very good ones. I think the addition of "destroysnap" or "destroyclone" are particularly good because they could be added without conflicting with or changing the existing interfaces. benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] New zfs pr0n server :)))
Diego Righi wrote: Hi all, I just built a new zfs server for home and, being a long time and avid reader of this forum, I'm going to post my config specs and my benchmarks hoping this could be of some help for others :) http://www.sickness.it/zfspr0nserver.jpg http://www.sickness.it/zfspr0nserver.txt http://www.sickness.it/zfspr0nserver.png http://www.sickness.it/zfspr0nserver.pdf Correct me if I'm wrong: from the benchmark results, I understand that this setup is slow at writing, but fast at reading (and this is perfect for my usage, copying large files once and then accessing only to read them). It also seems that at 128kb it gives the best performances, iirc due to the zfs stripe size (again, correct me if I'm wrong :). I'd happily try any other test, but if you suggest bonnie++ please tell me what's the right version to use, too much of them I really can't understand which to try! tnx :) Classy. +1 for style. ;) benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZVol Panic on 62
May 25 23:32:59 summer unix: [ID 836849 kern.notice] May 25 23:32:59 summer ^Mpanic[cpu1]/thread=1bf2e740: May 25 23:32:59 summer genunix: [ID 335743 kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ff00232c3a80 addr=490 occurred in module "unix" due to a NULL pointer dereference May 25 23:32:59 summer unix: [ID 10 kern.notice] May 25 23:32:59 summer unix: [ID 839527 kern.notice] grep: May 25 23:32:59 summer unix: [ID 753105 kern.notice] #pf Page fault May 25 23:32:59 summer unix: [ID 532287 kern.notice] Bad kernel fault at addr=0x490 May 25 23:32:59 summer unix: [ID 243837 kern.notice] pid=18425, pc=0xfb83b6bb, sp=0xff00232c3b78, eflags=0x10246 May 25 23:32:59 summer unix: [ID 211416 kern.notice] cr0: 8005003b cr4: 6f8 May 25 23:32:59 summer unix: [ID 354241 kern.notice] cr2: 490 cr3: 1fce52000 cr8: c May 25 23:32:59 summer unix: [ID 592667 kern.notice]rdi: 490 rsi:0 rdx: 1bf2e740 May 25 23:32:59 summer unix: [ID 592667 kern.notice]rcx:0 r8:d r9: 62ccc700 May 25 23:32:59 summer unix: [ID 592667 kern.notice]rax:0 rbx:0 rbp: ff00232c3bd0 May 25 23:32:59 summer unix: [ID 592667 kern.notice]r10: fc18 r11:0 r12: 490 May 25 23:32:59 summer unix: [ID 592667 kern.notice]r13: 450 r14: 52e3aac0 r15:0 May 25 23:32:59 summer unix: [ID 592667 kern.notice]fsb:0 gsb: fffec3731800 ds: 4b May 25 23:32:59 summer unix: [ID 592667 kern.notice] es: 4b fs:0 gs: 1c3 May 25 23:33:00 summer unix: [ID 592667 kern.notice]trp:e err:2 rip: fb83b6bb May 25 23:33:00 summer unix: [ID 592667 kern.notice] cs: 30 rfl:10246 rsp: ff00232c3b78 May 25 23:33:00 summer unix: [ID 266532 kern.notice] ss: 38 May 25 23:33:00 summer unix: [ID 10 kern.notice] May 25 23:33:00 summer genunix: [ID 655072 kern.notice] ff00232c3960 unix:die+c8 () May 25 23:33:00 summer genunix: [ID 655072 kern.notice] ff00232c3a70 unix:trap+135b () May 25 23:33:00 summer genunix: [ID 655072 kern.notice] ff00232c3a80 unix:cmntrap+e9 () May 25 23:33:00 summer genunix: [ID 655072 kern.notice] ff00232c3bd0 unix:mutex_enter+b () May 25 23:33:00 summer genunix: [ID 655072 kern.notice] ff00232c3c20 zfs:zvol_read+51 () May 25 23:33:00 summer genunix: [ID 655072 kern.notice] ff00232c3c50 genunix:cdev_read+3c () May 25 23:33:00 summer genunix: [ID 655072 kern.notice] ff00232c3cd0 specfs:spec_read+276 () May 25 23:33:00 summer genunix: [ID 655072 kern.notice] ff00232c3d40 genunix:fop_read+3f () May 25 23:33:00 summer genunix: [ID 655072 kern.notice] ff00232c3e90 genunix:read+288 () May 25 23:33:00 summer genunix: [ID 655072 kern.notice] ff00232c3ec0 genunix:read32+1e () May 25 23:33:00 summer genunix: [ID 655072 kern.notice] ff00232c3f10 unix:brand_sys_syscall32+1a3 () May 25 23:33:00 summer unix: [ID 10 kern.notice] May 25 23:33:00 summer genunix: [ID 672855 kern.notice] syncing file systems... Does anyone have an idea of what bug this might be? Occurred on X86 B62. I'm not seeing any putbacks into 63 or bugs that seem to match. Any insight is appreciated. Core's are available. benr. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS Send/RECV
I'm trying to test an install of ZFS to see if I can backup data from one machine to another. I'm using Solaris 5.10 on two VMware installs. When I do the zfs send | ssh zfs recv part, the file system (folder) is getting created, but none of the data that I have in my snapshot is sent. I can browse on the source machine to view the snapshot data pool/.zfs/snapshot/snap-name and I see the data. Am I missing something to make it copy all of the data? This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS, iSCSI + Mac OS X Tiger (globalSAN iSCSI)
George wrote: > I have set up an iSCSI ZFS target that seems to connect properly from > the Microsoft Windows initiator in that I can see the volume in MMC > Disk Management. > > > When I shift over to Mac OS X Tiger with globalSAN iSCSI, I am able to > set up the Targets with the target name shown by `iscsitadm list > target` and when I actually connect or "Log On" I see that one > connection exists on the Solaris server. I then go on to the Sessions > tab in globalSAN and I see the session details and it appears that > data is being transferred via the PDUs Sent, PDUs Received, Bytes, > etc. HOWEVER the connection then appears to terminate on the Solaris > side if I check it a few minutes later it shows no connections, but > the Mac OS X initiator still shows connected although no more traffic > appears to be flowing in the Session Statistics dialog area. > > > Additionally, when I then disconnect the Mac OS X initiator it seems > to drop fine on the Mac OS X side, even though the Solaris side has > shown it gone for a while, however when I reconnect or Log On again, > it seems to spin infinitely on the "Target Connect..." dialog. > Solaris is, interestingly, showing 1 connection while this apparent > issue (spinning beachball of death) is going on with globalSAN. Even > killing the Mac OS X process doesn't seem to get me full control again > as I have to restart the system to kill all processes (unless I can > hunt them down and `kill -9` them which I've not successfully done > thus far). > > Has anyone dealt with this before and perhaps be able to assist or at > least throw some further information towards me to troubleshoot this? When I learned of the globalSAN Initiator I was overcome with joy. after about 2 days of spending way too much time with it I gave up. Have a look at their forum (http://www.snsforums.com/index.php?s=b0c9031ebe1a89a40cfe4c417e3443f1&showforum=14). There are a wide range of problems. In my case connections to the target (Solaris/ZFS/iscsitgt) look fine and dandy initially, but you can use the connection, on reboot globalSAN goes psycho, etc. At this point I've given up on the product; at least for now. If I could actually get an accessable disk at least part of the time I'd dig my fingers into it, but it doesn't offer a usable remote disk to begin with and in a variety of other environments it have identical problems. I consider debugging it to be purely academic at this point. Its a great way to gain insight into the inner workings of iSCSI, but without source code or DTrace on the Mac its hard to expect any big gains. Thats my personal take. If you really wanna go hacking on it regardless bring it up on the Storage list and we can corporately enjoy the academic challenge of finding the problems, but there is nothing to suggest its an OpenSolaris issue. benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is there _any_ suitable motherboard?
I've just purchased an Asus P5K WS, which seems to work OK. I had to download the Marvell Yukon ethernet driver - but it's all working fine. It's also got a PCI-X slot - so I have one of those Super Micro 8 port SATA cards - providing a total of 16 SATA ports across the system. Other specs are one of those Intel E6750 1333MHz FSB CPUs and 2Gb of matched memory. Ben. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] System hang caused by a "bad" snapshot
> > > Hello Matthew, > > > Tuesday, September 12, 2006, 7:57:45 PM, you > > wrote: > > > MA> Ben Miller wrote: > > > >> I had a strange ZFS problem this morning. > The > > entire system would > > >> hang when mounting the ZFS filesystems. After > > trial and error I > > >> determined that the problem was with one of > the > > 2500 ZFS filesystems. > > >> When mounting that users' home the system > would > > hang and need to be > > >> rebooted. After I removed the snapshots (9 of > > them) for that > > >> filesystem everything was fine. > > >> > > >> I don't know how to reproduce this and didn't > get > > a crash dump. I > > >> don't remember seeing anything about this > before > > so I wanted to > > >> report it and see if anyone has any ideas. > > > > MA> Hmm, that sounds pretty bizarre, since I > don't > > think that mounting a > > MA> filesystem doesn't really interact with > snapshots > > at all. > > MA> Unfortunately, I don't think we'll be able to > > diagnose this without a > > MA> crash dump or reproducibility. If it happens > > again, force a crash dump > > MA> while the system is hung and we can take a > look > > at it. > > > > Maybe it wasn't hung after all. I've seen similar > > behavior here > > sometimes. Did your disks used in a pool were > > actually working? > > > > There was lots of activity on the disks (iostat and > status LEDs) until it got to this one filesystem > and > everything stopped. 'zpool iostat 5' stopped > running, the shell wouldn't respond and activity on > the disks stopped. This fs is relatively small >(175M used of a 512M quota). > Sometimes it takes a lot of time (30-50minutes) to > > mount a file system > > - it's rare, but it happens. And during this ZFS > > reads from those > > disks in a pool. I did report it here some time > ago. > > > In my case the system crashed during the evening > and it was left hung up when I came in during the > morning, so it was hung for a good 9-10 hours. > > The problem happened again last night, but for a > different users' filesystem. I took a crash dump > with it hung and the back trace looks like this: > > ::status > debugging crash dump vmcore.0 (64-bit) from hostname > operating system: 5.11 snv_40 (sun4u) > panic message: sync initiated > dump content: kernel pages only > > ::stack > 0xf0046a3c(f005a4d8, 2a100047818, 181d010, 18378a8, > 1849000, f005a4d8) > prom_enter_mon+0x24(2, 183c000, 18b7000, 2a100046c61, > 1812158, 181b4c8) > debug_enter+0x110(0, a, a, 180fc00, 0, 183e000) > abort_seq_softintr+0x8c(180fc00, 18abc00, 180c000, > 2a100047d98, 1, 1859800) > intr_thread+0x170(600019de0e0, 0, 6000d7bfc98, > 600019de110, 600019de110, > 600019de110) > zfs_delete_thread_target+8(600019de080, > , 0, 600019de080, > 6000d791ae8, 60001aed428) > zfs_delete_thread+0x164(600019de080, 6000d7bfc88, 1, > 2a100c4faca, 2a100c4fac8, > 600019de0e0) > thread_start+4(600019de080, 0, 0, 0, 0, 0) > > In single user I set the mountpoint for that user to > be none and then brought the system up fine. Then I > destroyed the snapshots for that user and their > filesystem mounted fine. In this case the quota was > reached with the snapshots and 52% used without. > > Ben Hate to re-open something from a year ago, but we just had this problem happen again. We have been running Solaris 10u3 on this system for awhile. I searched the bug reports, but couldn't find anything on this. I also think I understand what happened a little more. We take snapshots at noon and the system hung up during that time. When trying to reboot the system would hang on the ZFS mounts. After I boot into single use and remove the snapshot from the filesystem causing the problem everything is fine. The filesystem in question at 100% use with snapshots in use. Here's the back trace for the system when it was hung: > ::stack 0xf0046a3c(f005a4d8, 2a10004f828, 0, 181c850, 1848400, f005a4d8) prom_enter_mon+0x24(0, 0, 183b400, 1, 1812140, 181ae60) debug_enter+0x118(0, a, a, 180fc00, 0, 183d400) abort_seq_softintr+0x94(180fc00, 18a9800, 180c000, 2a10004fd98, 1, 1857c00) intr_thread+0x170(2, 30007b64bc0, 0, c001ed9, 110, 6000240) 0x985c8(300adca4c40, 0, 0, 0, 0, 30007b64bc0) dbuf_hold_impl+0x28(60008cd02e8, 0, 0, 0, 7b648d73, 2a105bb57c8) dbuf_hold_level+0x18(60008cd02e8, 0,
Re: [zfs-discuss] When I stab myself with this knife, it hurts... But - should it kill me?
Dick Davies wrote: > On 04/10/2007, Nathan Kroenert <[EMAIL PROTECTED]> wrote: > > >> Client A >> - import pool make couple-o-changes >> >> Client B >> - import pool -f (heh) >> > > >> Oct 4 15:03:12 fozzie ^Mpanic[cpu0]/thread=ff0002b51c80: >> Oct 4 15:03:12 fozzie genunix: [ID 603766 kern.notice] assertion >> failed: dmu_read(os, smo->smo_object, offset, size, entry_map) == 0 (0x5 >> == 0x0) >> , file: ../../common/fs/zfs/space_map.c, line: 339 >> Oct 4 15:03:12 fozzie unix: [ID 10 kern.notice] >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51160 >> genunix:assfail3+b9 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51200 >> zfs:space_map_load+2ef () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51240 >> zfs:metaslab_activate+66 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51300 >> zfs:metaslab_group_alloc+24e () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b513d0 >> zfs:metaslab_alloc_dva+192 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51470 >> zfs:metaslab_alloc+82 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b514c0 >> zfs:zio_dva_allocate+68 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b514e0 >> zfs:zio_next_stage+b3 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51510 >> zfs:zio_checksum_generate+6e () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51530 >> zfs:zio_next_stage+b3 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b515a0 >> zfs:zio_write_compress+239 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b515c0 >> zfs:zio_next_stage+b3 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51610 >> zfs:zio_wait_for_children+5d () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51630 >> zfs:zio_wait_children_ready+20 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51650 >> zfs:zio_next_stage_async+bb () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51670 >> zfs:zio_nowait+11 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51960 >> zfs:dbuf_sync_leaf+1ac () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b519a0 >> zfs:dbuf_sync_list+51 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51a10 >> zfs:dnode_sync+23b () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51a50 >> zfs:dmu_objset_sync_dnodes+55 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51ad0 >> zfs:dmu_objset_sync+13d () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51b40 >> zfs:dsl_pool_sync+199 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51bd0 >> zfs:spa_sync+1c5 () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51c60 >> zfs:txg_sync_thread+19a () >> Oct 4 15:03:12 fozzie genunix: [ID 655072 kern.notice] ff0002b51c70 >> unix:thread_start+8 () >> Oct 4 15:03:12 fozzie unix: [ID 10 kern.notice] >> > > >> Is this a known issue, already fixed in a later build, or should I bug it? >> > > It shouldn't panic the machine, no. I'd raise a bug. > > >> After spending a little time playing with iscsi, I have to say it's >> almost inevitable that someone is going to do this by accident and panic >> a big box for what I see as no good reason. (though I'm happy to be >> educated... ;) >> > > You use ACLs and TPGT groups to ensure 2 hosts can't simultaneously > access the same LUN by accident. You'd have the same problem with > Fibre Channel SANs. > I ran into similar problems when replicating via AVS. benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for OSX - it'll be in there.
Dale Ghent wrote: > ...and eventually in a read-write capacity: > > http://www.macrumors.com/2007/10/04/apple-seeds-zfs-read-write- > developer-preview-1-1-for-leopard/ > > Apple has seeded version 1.1 of ZFS (Zettabyte File System) for Mac > OS X to Developers this week. The preview updates a previous build > released on June 26, 2007. > Y! Finally my USB Thumb Drives will work on my MacBook! :) I wonder if it'll automatically mount the Zpool on my iPod when I sync it. benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS Quota Oddness
I've run across an odd issue with ZFS Quota's. This is an snv_43 system with several zones/zfs datasets, but only one effected. The dataset shows 10GB used, 12GB refered but when counting the files only has 6.7GB of data: zones/ABC10.8G 26.2G 12.0G /zones/ABC zones/[EMAIL PROTECTED]14.7M - 12.0G - [xxx:/zones/ABC/.zfs/snapshot/now] root# gdu --max-depth=1 -h . 43k ./dev 6.7G./root 1.5k./lu 6.7G. I don't understand what might the cause this disparity. This is an older box, snv_43. Any bugs that might apply, fixed or in progress? Thanks. benr. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Panic on Zpool Import (Urgent)
Today, suddenly, without any apparent reason that I can find, I'm getting panic's during zpool import. The system paniced earlier today and has been suffering since. This is snv_43 on a thumper. Here's the stack: panic[cpu0]/thread=99adbac0: assertion failed: ss != NULL, file: ../../common/fs/zfs/space_map.c, line: 145 fe8000a240a0 genunix:assfail+83 () fe8000a24130 zfs:space_map_remove+1d6 () fe8000a24180 zfs:space_map_claim+49 () fe8000a241e0 zfs:metaslab_claim_dva+130 () fe8000a24240 zfs:metaslab_claim+94 () fe8000a24270 zfs:zio_dva_claim+27 () fe8000a24290 zfs:zio_next_stage+6b () fe8000a242b0 zfs:zio_gang_pipeline+33 () fe8000a242d0 zfs:zio_next_stage+6b () fe8000a24320 zfs:zio_wait_for_children+67 () fe8000a24340 zfs:zio_wait_children_ready+22 () fe8000a24360 zfs:zio_next_stage_async+c9 () fe8000a243a0 zfs:zio_wait+33 () fe8000a243f0 zfs:zil_claim_log_block+69 () fe8000a24520 zfs:zil_parse+ec () fe8000a24570 zfs:zil_claim+9a () fe8000a24750 zfs:dmu_objset_find+2cc () fe8000a24930 zfs:dmu_objset_find+fc () fe8000a24b10 zfs:dmu_objset_find+fc () fe8000a24bb0 zfs:spa_load+67b () fe8000a24c20 zfs:spa_import+a0 () fe8000a24c60 zfs:zfs_ioc_pool_import+79 () fe8000a24ce0 zfs:zfsdev_ioctl+135 () fe8000a24d20 genunix:cdev_ioctl+55 () fe8000a24d60 specfs:spec_ioctl+99 () fe8000a24dc0 genunix:fop_ioctl+3b () fe8000a24ec0 genunix:ioctl+180 () fe8000a24f10 unix:sys_syscall32+101 () syncing file systems... done This is almost identical to a post to this list over a year ago titled "ZFS Panic". There was follow up on it but the results didn't make it back to the list. I spent time doing a full sweep for any hardware failures, pulled 2 drives that I suspected as problematic but weren't flagged as such, etc, etc, etc. Nothing helps. Bill suggested a 'zpool import -o ro' on the other post, but thats not working either. I _can_ use 'zpool import' to see the pool, but I have to force the import. A simple 'zpool import' returns output in about a minute. 'zpool import -f poolname' takes almost exactly 10 minutes every single time, like it hits some timeout and then panics. I did notice that while the 'zpool import' is running 'iostat' is useless, just hangs. I still want to believe this is some device misbehaving but I have no evidence to support that theory. Any and all suggestions are greatly appreciated. I've put around 8 hours into this so far and I'm getting absolutely nowhere. Thanks benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Removing An Errant Drive From Zpool
I made a really stupid mistake... having trouble removing a hot spare marked as failed I was trying several ways to put it back in a good state. One means I tried was to 'zpool add pool c5t3d0'... but I forgot to use the proper syntax "zpool add pool spare c5t3d0". Now I'm in a bind. I've got 4 large raidz2's and now this punty 500GB drive in the config: ... raidz2ONLINE 0 0 0 c5t7d0 ONLINE 0 0 0 c5t2d0 ONLINE 0 0 0 c7t7d0 ONLINE 0 0 0 c6t7d0 ONLINE 0 0 0 c1t7d0 ONLINE 0 0 0 c0t7d0 ONLINE 0 0 0 c4t3d0 ONLINE 0 0 0 c7t3d0 ONLINE 0 0 0 c6t3d0 ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 c0t3d0 ONLINE 0 0 0 c5t3d0ONLINE 0 0 0 spares c5t3d0FAULTED corrupted data c4t7d0AVAIL ... Detach and Remove won't work. Does anyone know of a way to get that c5t3d0 out of the data configuration and back to hot-spare where it belongs? However if I understand the layout properly, this should not have an adverse impact on my existing configuration I think. If I can't dump it, what happens when that disk fills up? I can't believe I made such a bone headed mistake. This is one of those times when a "Are you sure you...?" would be helpful. :( benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Removing An Errant Drive From Zpool
Eric Schrock wrote: > There's really no way to recover from this, since we don't have device > removal. However, I'm suprised that no warning was given. There are at > least two things that should have happened: > > 1. zpool(1M) should have warned you that the redundancy level you were >attempting did not match that of your existing pool. This doesn't >apply if you already have a mixed level of redundancy. > > 2. zpool(1M) should have warned you that the device was in use as an >active spare and not let you continue. > > What bits were you running? > snv_78, however the pool was created on snv_43 and hasn't yet been upgraded. Though, programatically, I can't see why there would be a difference in the way 'zpool' would handle the check. The big question is, if I'm stuck like the permanently, whats the potential risk? Could I potentially just fail that drive and leave it in a failed state? benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Removing An Errant Drive From Zpool
Robert Milkowski wrote: > If you can't re-create a pool (+backup&restore your data) I would > recommend to wait for device removal in zfs and in a mean time I would > attach another drive to it so you've got mirrored configuration and > remove them once there's a device removal. Since you're already > working on nevada you probably could adopt new bits quickly. > > The only question is - when device removal is going to be integrated - > last time someone mentioned it here it was supposed to be by the end > of last year... > Ya, I'm afraid your right. benr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Panic on Zpool Import (Urgent)
The solution here was to upgrade to snv_78. By "upgrade" I mean re-jumpstart the system. I tested snv_67 via net-boot but the pool paniced just as below. I also attempted using zfs_recover without success. I then tested snv_78 via net-boot, used both "aok=1" and "zfs:zfs_recover=1" and was able to (slowly) import the pool. Following that test I exported and then did a full re-install of the box. A very important note to anyone upgrading a Thumper! Don't forget about the NCQ bug. After upgrading to a release more recent than snv_60 add the following to /etc/system: set sata:sata_max_queue_depth = 0x1 If you don't life will be highly unpleasant and you'll believe that disks are failing everywhere when in fact they are not. benr. Ben Rockwood wrote: > Today, suddenly, without any apparent reason that I can find, I'm > getting panic's during zpool import. The system paniced earlier today > and has been suffering since. This is snv_43 on a thumper. Here's the > stack: > > panic[cpu0]/thread=99adbac0: assertion failed: ss != NULL, file: > ../../common/fs/zfs/space_map.c, line: 145 > > fe8000a240a0 genunix:assfail+83 () > fe8000a24130 zfs:space_map_remove+1d6 () > fe8000a24180 zfs:space_map_claim+49 () > fe8000a241e0 zfs:metaslab_claim_dva+130 () > fe8000a24240 zfs:metaslab_claim+94 () > fe8000a24270 zfs:zio_dva_claim+27 () > fe8000a24290 zfs:zio_next_stage+6b () > fe8000a242b0 zfs:zio_gang_pipeline+33 () > fe8000a242d0 zfs:zio_next_stage+6b () > fe8000a24320 zfs:zio_wait_for_children+67 () > fe8000a24340 zfs:zio_wait_children_ready+22 () > fe8000a24360 zfs:zio_next_stage_async+c9 () > fe8000a243a0 zfs:zio_wait+33 () > fe8000a243f0 zfs:zil_claim_log_block+69 () > fe8000a24520 zfs:zil_parse+ec () > fe8000a24570 zfs:zil_claim+9a () > fe8000a24750 zfs:dmu_objset_find+2cc () > fe8000a24930 zfs:dmu_objset_find+fc () > fe8000a24b10 zfs:dmu_objset_find+fc () > fe8000a24bb0 zfs:spa_load+67b () > fe8000a24c20 zfs:spa_import+a0 () > fe8000a24c60 zfs:zfs_ioc_pool_import+79 () > fe8000a24ce0 zfs:zfsdev_ioctl+135 () > fe8000a24d20 genunix:cdev_ioctl+55 () > fe8000a24d60 specfs:spec_ioctl+99 () > fe8000a24dc0 genunix:fop_ioctl+3b () > fe8000a24ec0 genunix:ioctl+180 () > fe8000a24f10 unix:sys_syscall32+101 () > > syncing file systems... done > > This is almost identical to a post to this list over a year ago titled > "ZFS Panic". There was follow up on it but the results didn't make it > back to the list. > > I spent time doing a full sweep for any hardware failures, pulled 2 > drives that I suspected as problematic but weren't flagged as such, etc, > etc, etc. Nothing helps. > > Bill suggested a 'zpool import -o ro' on the other post, but thats not > working either. > > I _can_ use 'zpool import' to see the pool, but I have to force the > import. A simple 'zpool import' returns output in about a minute. > 'zpool import -f poolname' takes almost exactly 10 minutes every single > time, like it hits some timeout and then panics. > > I did notice that while the 'zpool import' is running 'iostat' is > useless, just hangs. I still want to believe this is some device > misbehaving but I have no evidence to support that theory. > > Any and all suggestions are greatly appreciated. I've put around 8 > hours into this so far and I'm getting absolutely nowhere. > > Thanks > > benr. > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] zpool status -x strangeness on b78
We run a cron job that does a 'zpool status -x' to check for any degraded pools. We just happened to find a pool degraded this morning by running 'zpool status' by hand and were surprised that it was degraded as we didn't get a notice from the cron job. # uname -srvp SunOS 5.11 snv_78 i386 # zpool status -x all pools are healthy # zpool status pool1 pool: pool1 state: DEGRADED scrub: none requested config: NAME STATE READ WRITE CKSUM pool1DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 c1t8d0 REMOVED 0 0 0 c1t9d0 ONLINE 0 0 0 c1t10d0 ONLINE 0 0 0 c1t11d0 ONLINE 0 0 0 errors: No known data errors I'm going to look into it now why the disk is listed as removed. Does this look like a bug with 'zpool status -x'? Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] List of ZFS patches to be released with Solaris 10 U5
Experts, Do you know where I could find the list of all the ZFS patches that will be released with Solaris 10 U5? My customer told me that they've seen such list for prior update releases. I've not been able to find anything like it in the usual places. TIA, Ben ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Clearing corrupted file errors
Hi, Sorry if this is a RTM issue - but I wanted to be sure before continuing. I received a corrupted file error on one of my pools. I removed the file, and the status command now shows the following: > zpool status -v rpool pool: rpool state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: none requested config: NAMESTATE READ WRITE CKSUM rpool ONLINE 0 0 0 raidz1ONLINE 0 0 0 c0t0d0 ONLINE 0 0 0 c0t1d0 ONLINE 0 0 0 c0t2d0 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: rpool/duke:<0x8237> I tried running "zpool clear rpool" to clear the error, but it persists in the status output. Should a "zpool scrub rpool" get rid of this error? Thanks, Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Cannot delete errored file
Hi, I can't seem to delete a file in my zpool that has permanent errors: zpool status -vx pool: rpool state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: scrub completed after 2h10m with 1 errors on Tue Jun 3 11:36:49 2008 config: NAMESTATE READ WRITE CKSUM rpool ONLINE 0 0 0 raidz1ONLINE 0 0 0 c0t0d0 ONLINE 0 0 0 c0t1d0 ONLINE 0 0 0 c0t2d0 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /export/duke/test/Acoustic/3466/88832/09 - Check.mp3 rm "/export/duke/test/Acoustic/3466/88832/09 - Check.mp3" rm: cannot remove `/export/duke/test/Acoustic/3466/88832/09 - Check.mp3': I/O error Each time I try to do anything to the file, the checksum error count goes up on the pool. I also tried a mv and a cp over the top - but same I/O error. I performed a "zpool scrub rpool" followed by a "zpool clear rpool" - but still get the same error. Any ideas? PS - I'm running snv_86, and use the sata driver on an intel x86 architecture. B This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Cannot delete errored file
Hello again, I'm not making progress on this. Every time I run a zpool scrub rpool I see: $ zpool status -vx pool: rpool state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: scrub in progress for 0h0m, 0.01% done, 177h43m to go config: NAMESTATE READ WRITE CKSUM rpool DEGRADED 0 0 8 raidz1DEGRADED 0 0 8 c0t0d0 DEGRADED 0 0 0 too many errors c0t1d0 ONLINE 0 0 0 c0t2d0 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /export/duke/test/Acoustic/3466/88832/09 - Check.mp3 I popped in a brand new disk of the same size, and did a zpool replace on the persistently degraded drive and the new drive. i.e.: $ zpool replace rpool c0t0d0 c0t7d0 But that simply had the effect of transferring the issue to the new drive: $ zpool status -xv rpool pool: rpool state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver completed after 2h41m with 1 errors on Wed Jun 4 20:22:27 2008 config: NAME STATE READ WRITE CKSUM rpool DEGRADED 0 0 8 raidz1 DEGRADED 0 0 8 spare DEGRADED 0 0 0 c0t0d0 DEGRADED 0 0 0 too many errors c0t7d0 ONLINE 0 0 0 c0t1d0ONLINE 0 0 0 c0t2d0ONLINE 0 0 0 spares c0t7d0 INUSE currently in use $ zpool detach rpool c0t0d0 $ zpool status -vx rpool pool: rpool state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver completed after 2h41m with 1 errors on Wed Jun 4 20:22:27 2008 config: NAMESTATE READ WRITE CKSUM rpool ONLINE 0 0 8 raidz1ONLINE 0 0 8 c0t7d0 ONLINE 0 0 0 c0t1d0 ONLINE 0 0 0 c0t2d0 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: <0xc3>:<0x1c0> $ zpool scrub rpool ... $ zpool status -vx rpool pool: rpool state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: scrub in progress for 0h0m, 0.00% done, 0h0m to go config: NAMESTATE READ WRITE CKSUM rpool DEGRADED 0 0 4 raidz1DEGRADED 0 0 4 c0t7d0 DEGRADED 0 0 0 too many errors c0t1d0 ONLINE 0 0 0 c0t2d0 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /export/duke/test/Acoustic/3466/88832/09 - Check.mp3 $ rm -f "/export/duke/test/Acoustic/3466/88832/09 - Check.mp3" rm: cannot remove `/export/duke/test/Acoustic/3466/88832/09 - Check.mp3': I/O error I'm guessing this isn't a hardware fault, but a glitch in ZFS - but am hoping to be proved wrong. Any ideas before I rebuild the pool from scratch? And if I do, is there anything I can do to prevent this problem in the future? B This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Cannot delete errored file
Hi Marc, $ : > 09 - Check.mp3 bash: 09 - Check.mp3: I/O error $ cd .. $ rm -rf BAD $ rm: cannot remove `BAD/09 - Check.mp3': I/O error I'll try shuffling the cables - but as you see above it occasionally reports on a different disk - so imagine the cables are OK. Also, the new disk I added has a new cable too, and on a different SATA port - which is also showing up as degraded. Is there any lower level debugging that I can enable to try and work out what is going on. This machine has been running fine since last August. I couldn't see anything in builds later than snv 86 that might help - but I could try upgrading to the latest? B This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS and ACL's over NFSv3
Can someone please clarify the ability to utilize ACL's over NFSv3 from a ZFS share? I can "getfacl" but I can't "setfacl". I can't find any documentation in this regard. My suspicion is that that ZFS Shares must be NFSv4 in order to utilize ACLs but I'm hoping this isn't the case. Can anyone definitively speak to this? The closest related bug I can find is 6340720 which simply says "See comments." benr. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Cannot delete errored file
Thanks Marc - I'll run memtest on Monday, and re-seat memory/cpu//cards etc. If that fails, I'll try moving the devices onto a different SATA controller. Failing that I'll rebuild from scratch. Failing that, I'll get a new motherboard! Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Cannot delete errored file
Hi, Today's update: - I ran a memtest a few times - no errors. - I reseated, re-routed ad switched all connectors/cables - I'm currently running a scrub, but it's showing vast numbers of cksum errors now across all devices: $ zpool status -v pool: rpool state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: scrub in progress for 0h5m, 3.35% done, 2h26m to go config: NAMESTATE READ WRITE CKSUM rpool DEGRADED 0 0 211K raidz1DEGRADED 0 0 211K c0t7d0 DEGRADED 0 0 0 too many errors c0t1d0 DEGRADED 0 0 0 too many errors c0t2d0 DEGRADED 0 0 0 too many errors errors: Permanent errors have been detected in the following files: /export/duke/test/Acoustic/3466/88832/09 - Check.mp3 I'll start moving each device over to a different controller to see if that helps once the scrub completes. Still getting I/O errors trying to delete that file. Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Cannot delete errored file
Hi Marc, Thanks for all of your suggestions. I'll restart memtest when I'm next in the office and leave it running overnight. I can recreate the pool - but I guess the question is am I safe to do this on the existing setup, or am I going to hit the same issue again sometime? Assuming I don't find any obvious hardware issues - wouldn't this be a regarded as flaw in ZFS (i.e. no way of clearing such an error without a rebuild)? Would I be safer rebuilding to a pair of mirrors rather than a 3 disk zraid + hotspare? Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Cannot delete errored file
Sent response by private message. Today's findings are that the cksum errors appear on the new disk on the other controller too - so I've ruled out controllers & cables. It's probably as Jeff says - just got to figure out now how to prove the memory is duff. Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Cannot delete errored file
Hi, It's an ASUS P5K-WS board with 2Gb of Corsair TwinX DDR2 8500 1066MHz Non-ECC memory. The board uses the Intel P35 chipset - it also will not support ECC RAM. TBH, this is probably the last time I'll get an ASUS, as this is the second board I've got through - the first one died for no particular reason. I've been recommended a Supermicro C2SBX Mobo - this will take my existing Supermicro PCI-X card as well as my current Intel CPU. The only problem is that it's DDR3 - and ECC DDR3 ram is pretty hard to come by right now. I'll still try a long memtest run, followed by a rebuild of the errored pool. I'll have a read around to see if there's anyway of making the memory more stable on this mobo. Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Cannot delete errored file
Hi, Quick update: I left memtest running over night - 39 passes, no errors. I also attempted to force the BIOS to run the memory at 800MHz & 5-5-5-15 as suggested - but the machine became very unstable - long boot times; PCI-Express failure of Yukon network card on booting etc. I've switched it back to Auto speed&timing for now. I'll just hope that it was a one-off glitch that corrupted the pool. I'm going to rebuild the pool this weekend. Thanks for all the suggestions. Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] J4200/J4400 Array
Hi, According to the Sun Handbook, there is a new array : SAS interface 12 disks SAS or SATA ZFS could be used nicely with this box. There is an another version called J4400 with 24 disks. Doc is here : http://docs.sun.com/app/docs/coll/j4200 Does someone know price and availability for these products ? Best Regards, Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] 40min ls in empty directory
I've run into an odd problem which I lovingly refer to as a "black hole
directory".
On a Thumper used for mail stores we've found find's take an exceptionally long
time to run. There are directories that have as many as 400,000 files, which I
immediately considered the culprit. However, under investigation, they aren't
the problem at all. The problem is seen here in this truss output (first
column is delta time):
0.0001 lstat64("tmp", 0x08046A20) = 0
0. openat(AT_FDCWD, "tmp", O_RDONLY|O_NDELAY|O_LARGEFILE) = 8
0.0001 fcntl(8, F_SETFD, 0x0001) = 0
0. fstat64(8, 0x08046920) = 0
0. fstat64(8, 0x08046AB0) = 0
0. fchdir(8) = 0
1321.3133 getdents64(8, 0xFEE48000, 8192) = 48
1255.8416 getdents64(8, 0xFEE48000, 8192) = 0
0.0001 fchdir(7) = 0
0.0001 close(8)= 0
These two getdents64 syscalls take approx 20 mins each. Notice that the
directory structure is 48 bytes, the directory is empty:
drwx-- 2 102 1022 Feb 21 02:24 tmp
My assumption is that the directory is corrupt, but I'd like to prove that. I
have a scrub running on the pool, but its got about 16 hours to go before it
completes. 20% complete thus far and nothing is reported.
No errors are logged when I stimulate this problem.
Does anyone have suggestions on how to get additional data on this issue? I've
used dtrace flows to examine, however what I really want to see is the zio's as
a result of the getdents, but can't see how to do so. Ideally I'd quiet the
system and watch all zio's occurring while I stimulate it, but this is
production and not possible. If anyone knows how to watch DMU/ZIO activity
that _only_ pertains to a certain PID please let me know. ;)
Suggestions on how to pro-actively catch these sorts of instances are welcome,
as are alternative explanations.
benr.
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] zvol Performance
Hello, I'm curious if anyone would mind sharing their experiences with zvol's. I recently started using zvol as an iSCSI backend and was supprised by the performance I was getting. Further testing revealed that it wasn't an iSCSI performance issue but a zvol issue. Testing on a SATA disk locally, I get these numbers (sequentual write): UFS: 38MB/s ZFS: 38MB/s Zvol UFS: 6MB/s Zvol Raw: ~6MB/s ZFS is nice and fast but Zvol performance just drops off a cliff. Suggestion or observations by others using zvol would be extremely helpful. My current testing is being done using a debug build of B44 (NV 6/10/06). benr. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Home Server with ZFS
Hi, I'm plan to build home server that will host my svn repository, fileserver, mailserver and webserver. This is my plan.. I have an old dell precision 420 dual 933Mhz pIII cpus. Inside this i have one scsi 9.1G hdd and 2 80G ide hdds. I am going to install solaris 10 on the scsi drive and have it as the boot disk. I will then create a zfs mirror on the two ide drives. Since i dont want to mix internet facing services (mailserver, webservers) with my internal services (svn server, fileserver) i am going to use zones to isolate them. Not sure how many zones just yet. In this configureation i hope too of gained the protection of having the serives mirrors ( will perform backups also ). What i dont know is what happens if the boot disk dies? can i replace is, install solaris again and get it to see the zfs mirror? Also what happens if one of the ide drives fails? can i plug another one in and run some zfs commands to make it part of the mirror? Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] System hang caused by a "bad" snapshot
I had a strange ZFS problem this morning. The entire system would hang when mounting the ZFS filesystems. After trial and error I determined that the problem was with one of the 2500 ZFS filesystems. When mounting that users' home the system would hang and need to be rebooted. After I removed the snapshots (9 of them) for that filesystem everything was fine. I don't know how to reproduce this and didn't get a crash dump. I don't remember seeing anything about this before so I wanted to report it and see if anyone has any ideas. The system is a Sun Fire 280R with 3GB of RAM running SXCR b40. The pool looks like this (I'm running a scrub currently): # zpool status pool1 pool: pool1 state: ONLINE scrub: scrub in progress, 78.61% done, 0h18m to go config: NAME STATE READ WRITE CKSUM pool1ONLINE 0 0 0 raidz ONLINE 0 0 0 c1t8d0 ONLINE 0 0 0 c1t9d0 ONLINE 0 0 0 c1t10d0 ONLINE 0 0 0 c1t11d0 ONLINE 0 0 0 errors: No known data errors Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Re[2]: System hang caused by a "bad" snapshot
> Hello Matthew, > Tuesday, September 12, 2006, 7:57:45 PM, you wrote: > MA> Ben Miller wrote: > >> I had a strange ZFS problem this morning. The > entire system would > >> hang when mounting the ZFS filesystems. After > trial and error I > >> determined that the problem was with one of the > 2500 ZFS filesystems. > >> When mounting that users' home the system would > hang and need to be > >> rebooted. After I removed the snapshots (9 of > them) for that > >> filesystem everything was fine. > >> > >> I don't know how to reproduce this and didn't get > a crash dump. I > >> don't remember seeing anything about this before > so I wanted to > >> report it and see if anyone has any ideas. > > MA> Hmm, that sounds pretty bizarre, since I don't > think that mounting a > MA> filesystem doesn't really interact with snapshots > at all. > MA> Unfortunately, I don't think we'll be able to > diagnose this without a > MA> crash dump or reproducibility. If it happens > again, force a crash dump > MA> while the system is hung and we can take a look > at it. > > Maybe it wasn't hung after all. I've seen similar > behavior here > sometimes. Did your disks used in a pool were > actually working? > There was lots of activity on the disks (iostat and status LEDs) until it got to this one filesystem and everything stopped. 'zpool iostat 5' stopped running, the shell wouldn't respond and activity on the disks stopped. This fs is relatively small (175M used of a 512M quota). > Sometimes it takes a lot of time (30-50minutes) to > mount a file system > - it's rare, but it happens. And during this ZFS > reads from those > disks in a pool. I did report it here some time ago. > In my case the system crashed during the evening and it was left hung up when I came in during the morning, so it was hung for a good 9-10 hours. Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Re[2]: System hang caused by a "bad" snapshot
> > Hello Matthew, > > Tuesday, September 12, 2006, 7:57:45 PM, you > wrote: > > MA> Ben Miller wrote: > > >> I had a strange ZFS problem this morning. The > > entire system would > > >> hang when mounting the ZFS filesystems. After > > trial and error I > > >> determined that the problem was with one of the > > 2500 ZFS filesystems. > > >> When mounting that users' home the system would > > hang and need to be > > >> rebooted. After I removed the snapshots (9 of > > them) for that > > >> filesystem everything was fine. > > >> > > >> I don't know how to reproduce this and didn't > get > > a crash dump. I > > >> don't remember seeing anything about this > before > > so I wanted to > > >> report it and see if anyone has any ideas. > > > > MA> Hmm, that sounds pretty bizarre, since I don't > > think that mounting a > > MA> filesystem doesn't really interact with > snapshots > > at all. > > MA> Unfortunately, I don't think we'll be able to > > diagnose this without a > > MA> crash dump or reproducibility. If it happens > > again, force a crash dump > > MA> while the system is hung and we can take a > look > > at it. > > > > Maybe it wasn't hung after all. I've seen similar > > behavior here > > sometimes. Did your disks used in a pool were > > actually working? > > > > There was lots of activity on the disks (iostat and > status LEDs) until it got to this one filesystem and > everything stopped. 'zpool iostat 5' stopped > running, the shell wouldn't respond and activity on > the disks stopped. This fs is relatively small > (175M used of a 512M quota). > > Sometimes it takes a lot of time (30-50minutes) to > > mount a file system > > - it's rare, but it happens. And during this ZFS > > reads from those > > disks in a pool. I did report it here some time > ago. > > > In my case the system crashed during the evening > and it was left hung up when I came in during the > morning, so it was hung for a good 9-10 hours. > The problem happened again last night, but for a different users' filesystem. I took a crash dump with it hung and the back trace looks like this: > ::status debugging crash dump vmcore.0 (64-bit) from hostname operating system: 5.11 snv_40 (sun4u) panic message: sync initiated dump content: kernel pages only > ::stack 0xf0046a3c(f005a4d8, 2a100047818, 181d010, 18378a8, 1849000, f005a4d8) prom_enter_mon+0x24(2, 183c000, 18b7000, 2a100046c61, 1812158, 181b4c8) debug_enter+0x110(0, a, a, 180fc00, 0, 183e000) abort_seq_softintr+0x8c(180fc00, 18abc00, 180c000, 2a100047d98, 1, 1859800) intr_thread+0x170(600019de0e0, 0, 6000d7bfc98, 600019de110, 600019de110, 600019de110) zfs_delete_thread_target+8(600019de080, , 0, 600019de080, 6000d791ae8, 60001aed428) zfs_delete_thread+0x164(600019de080, 6000d7bfc88, 1, 2a100c4faca, 2a100c4fac8, 600019de0e0) thread_start+4(600019de080, 0, 0, 0, 0, 0) In single user I set the mountpoint for that user to be none and then brought the system up fine. Then I destroyed the snapshots for that user and their filesystem mounted fine. In this case the quota was reached with the snapshots and 52% used without. Ben This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
