Re: [zfs-discuss] Intermittent ZFS hang

2011-01-03 Thread Paul Armstrong
I'm not sure if they still apply to B134, but it seems similar to problems caused by transaction group issues in the past. Have you looked at the threads involving setting zfs:zfs_write_limit_override, zfs:zfs_vdev_max_pending or zfs:zfs_txg_timeout in /etc/system? Paul -- This message posted

Re: [zfs-discuss] Intermittent ZFS hang

2010-12-28 Thread Robin Axelsson
The HBA I use is an LSI MegaRAID 1038E-R but I guess it doesn't really matter as most OEM manufacturers such as Dell, Intel, HP, IBM use the LSI 1068e/1078e or the newer 2008e/2018e Megaraid chips which I believe use pretty much the same firmware. So I guess I could change these settings in the

Re: [zfs-discuss] Intermittent ZFS hang

2010-12-27 Thread Richard Elling
On Dec 21, 2010, at 4:41 PM, Robin Axelsson wrote: > There's nothing odd about the physical mounting of the hard drives. All > drives are firmly attached and secured in their casings, no loose connections > etc. There is some dust but not more than the hardware should be able to > handle. > >

Re: [zfs-discuss] Intermittent ZFS hang

2010-12-22 Thread Edward Ned Harvey
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Charles J. Knipe > > Some more information about our configuration: We're running OpenSolaris > svn-134. ZFS is at version 22. Our disks are 15kRPM 300gb Seagate Cheetahs, > mounted in Promi

Re: [zfs-discuss] Intermittent ZFS hang

2010-12-22 Thread Robin Axelsson
I didn't hot swap the drive but yes, the new drive is in the same "slot" as the old one was (i.e. using the same connector/channel on the fan out cable). What I did was that I turned off the system, and booted it up after disconnecting the physical drive that I suspected was c0t3d0. My guess was

Re: [zfs-discuss] Intermittent ZFS hang

2010-12-22 Thread Bob Friesenhahn
On Tue, 21 Dec 2010, Robin Axelsson wrote: There's nothing odd about the physical mounting of the hard drives. All drives are firmly attached and secured in their casings, no loose connections etc. There is some dust but not more than the hardware should be able to handle. I replaced the hard

Re: [zfs-discuss] Intermittent ZFS hang

2010-12-21 Thread Robin Axelsson
There's nothing odd about the physical mounting of the hard drives. All drives are firmly attached and secured in their casings, no loose connections etc. There is some dust but not more than the hardware should be able to handle. I replaced the hard drive with another one of the same size, I fi

Re: [zfs-discuss] Intermittent ZFS hang

2010-12-20 Thread Bob Friesenhahn
On Sun, 19 Dec 2010, Robin Axelsson wrote: To conclude this (in case you don't view this message using a monospace font) all drives in the affected storage pool (c9t0d0 - c9t7d0) report 2 Illegal Requests (save c9t3d0 that reports 5 illegal requests). There is one drive (c9t3d0) that looks li

Re: [zfs-discuss] Intermittent ZFS hang

2010-12-20 Thread Robin Axelsson
I have now upgraded to OpenIndiana b148 which should fix those bugs that you mentioned. I lost the picture on the monitor but by ssh:ing from another computer the system seems to be running fine. The problems have become worse now and I get a freeze every time I try to access the 8-disk raidz2

Re: [zfs-discuss] Intermittent ZFS hang

2010-09-28 Thread Robin Axelsson
I am using a zpool for swap that is located in the rpool (i.e. not in the storage pool). The system disk contains four primary partitions where the first contains the system volume (c7d0s0) two are windows partitions (c7d0p2 and c7d0p3) and the fourth (c7d0p4) is a zfs pool dedicated for Virtual

Re: [zfs-discuss] Intermittent ZFS hang

2010-09-28 Thread Robin Axelsson
I have now run some hardware tests as suggested by Cindy.'iostat -En' indicates no errors, i.e. after carefully checking the output from this command, all errors are followed by zeroes. The only messages found in /var/adm/messages are the following: opensolaris scsi: [ID 365881 kern.info] /p.

Re: [zfs-discuss] Intermittent ZFS hang

2010-09-28 Thread James C. McPherson
On 28/09/10 09:22 PM, Robin Axelsson wrote: I also have this problem on my system which consists of an AMD Phenom 2 X4 with system pools on various hard drives connected to the SB750 controller and a larger raidz2 storage pool connected to an LSI 1068e controller (using IT mode). The storage pool

Re: [zfs-discuss] Intermittent ZFS hang

2010-09-28 Thread Robin Axelsson
I also have this problem on my system which consists of an AMD Phenom 2 X4 with system pools on various hard drives connected to the SB750 controller and a larger raidz2 storage pool connected to an LSI 1068e controller (using IT mode). The storage pool is also used to share files using CIFS. Th

Re: [zfs-discuss] Intermittent ZFS hang

2010-09-27 Thread Jason J. W. Williams
If one was sticking with OpenSolaris for the short term, is something older than 134 more stable/less buggy? Not using de-dupe. -J On Thu, Sep 23, 2010 at 6:04 PM, Richard Elling wrote: > Hi Charles, > There are quite a few bugs in b134 that can lead to this. Alas, due to the > new > regime, the

Re: [zfs-discuss] Intermittent ZFS hang

2010-09-23 Thread Richard Elling
Hi Charles, There are quite a few bugs in b134 that can lead to this. Alas, due to the new regime, there was a period of time where the distributions were not being delivered. If I were in your shoes, I would upgrade to OpenIndiana b147 which has 26 weeks of maturity and bug fixes over b134. http:

Re: [zfs-discuss] Intermittent ZFS hang

2010-09-23 Thread Charles J. Knipe
So, I'm still having problems with intermittent hangs on write with my ZFS pool. Details from my original post are below. Since posting that, I've gone back and forth with a number of you, and gotten a lot of useful advice, but I'm still trying to get to the root of the problem so I can correc

Re: [zfs-discuss] Intermittent ZFS hang

2010-09-13 Thread Charles J. Knipe
> > At first we blamed de-dupe, but we've disabled that. Next we > suspected > > the SSD log disks, but we've seen the problem with those removed, as > > well. > > Did you have dedup enabled and then disabled it? If so, data can (or > will) be deduplicated on the drives. Currently the only way of

Re: [zfs-discuss] Intermittent ZFS hang

2010-09-13 Thread Roy Sigurd Karlsbakk
> At first we blamed de-dupe, but we've disabled that. Next we suspected > the SSD log disks, but we've seen the problem with those removed, as > well. Did you have dedup enabled and then disabled it? If so, data can (or will) be deduplicated on the drives. Currently the only way of de-deduping t

Re: [zfs-discuss] Intermittent ZFS hang

2010-09-13 Thread Charles J. Knipe
> > > Charles, > > Just like UNIX, there are several ways to drill down > on the problem.  I > would probably start with a live crash dump (savecore > -L) when you see > the problem.  Another method would be to grap > multiple "stats" commands > during the problem to see where you can drill down

Re: [zfs-discuss] Intermittent ZFS hang

2010-08-31 Thread Cindy Swearingen
Hi Charles, You might want rule out hardware issues first... You can review iostat -En or the /var/adm/messages file to see if any driver related error messages are related to the hangs, like this: c4t40d0 Soft Errors: 7 Hard Errors: 0 Transport Errors: 0 Vendor: SUN Product: Stor

Re: [zfs-discuss] Intermittent ZFS hang

2010-08-31 Thread David Blasingame Oracle
Charles, Just like UNIX, there are several ways to drill down on the problem. I would probably start with a live crash dump (savecore -L) when you see the problem. Another method would be to grap multiple "stats" commands during the problem to see where you can drill down later. I would pr

Re: [zfs-discuss] Intermittent ZFS hang

2010-08-30 Thread Charles J. Knipe
David, Thanks for your reply. Answers to your questions are below. > Is it just ZFS hanging (or what it appears to be is > slowing down or > blocking) or does the whole system hang?  Only the ZFS storage is affected. Any attempt to write to it blocks until the issue passes. Other than that

Re: [zfs-discuss] Intermittent ZFS hang

2010-08-30 Thread David Blasingame Oracle
Charles, Is it just ZFS hanging (or what it appears to be is slowing down or blocking) or does the whole system hang? A couple of questions What does iostat show during the time period of the slowdown? What does mpstat show during the time of the slowdown? You can look at the metadata stati

Re: [zfs-discuss] Intermittent ZFS hang

2010-08-30 Thread Remco Lengers
Charles, Did you check for any HW issues reported during the hangs? fmdump -ev and the like? ..Remco On 8/30/10 6:02 PM, Charles J. Knipe wrote: Howdy, We're having a ZFS performance issue over here that I was hoping you guys could help me troubleshoot. We have a ZFS pool made up of 24 d

[zfs-discuss] Intermittent ZFS hang

2010-08-30 Thread Charles J. Knipe
Howdy, We're having a ZFS performance issue over here that I was hoping you guys could help me troubleshoot. We have a ZFS pool made up of 24 disks, arranged into 7 raid-z devices of 4 disks each. We're using it as an iSCSI back-end for VMWare and some Oracle RAC clusters. Under normal circu