Re: Recurring problem: processes block accessing UFS file system

2006-01-12 Thread Don Lewis
On 12 Jan, Don Lewis wrote: > On 11 Jan, Denis Shaposhnikov wrote: >> Hi! >> >>> "Don" == Don Lewis <[EMAIL PROTECTED]> writes: >> >> Don> Are you using any unusual file systems, such as nullfs or >> Don> unionfs? >> >> >> Yes, I'm use a lots of nullfs. This is a host system for about 20

Re: Recurring problem: processes block accessing UFS file system

2006-01-12 Thread Don Lewis
On 11 Jan, Denis Shaposhnikov wrote: > Hi! > >> "Don" == Don Lewis <[EMAIL PROTECTED]> writes: > > Don> Are you using any unusual file systems, such as nullfs or > Don> unionfs? > > >> Yes, I'm use a lots of nullfs. This is a host system for about 20 > >> jails with nullfs mounted ro sys

Re: Recurring problem: processes block accessing UFS file system

2006-01-11 Thread Denis Shaposhnikov
Hi! > "Don" == Don Lewis <[EMAIL PROTECTED]> writes: Don> Are you using any unusual file systems, such as nullfs or Don> unionfs? >> Yes, I'm use a lots of nullfs. This is a host system for about 20 >> jails with nullfs mounted ro system: Don> That would be my guess as to the cause of

Re: Recurring problem: processes block accessing UFS file system

2006-01-05 Thread Don Lewis
On 5 Jan, Denis Shaposhnikov wrote: >> "Don" == Don Lewis <[EMAIL PROTECTED]> writes: > > Don> Are you using any unusual file systems, such as nullfs or > Don> unionfs? > > Yes, I'm use a lots of nullfs. This is a host system for about 20 > jails with nullfs mounted ro system: That would

Re: Recurring problem: processes block accessing UFS file system

2006-01-05 Thread Denis Shaposhnikov
> "Don" == Don Lewis <[EMAIL PROTECTED]> writes: Don> Are you using any unusual file systems, such as nullfs or Don> unionfs? Yes, I'm use a lots of nullfs. This is a host system for about 20 jails with nullfs mounted ro system: /dev/mirror/gm0s1a on / (ufs, local, read-only) devfs on /dev

Re: Recurring problem: processes block accessing UFS file system

2006-01-05 Thread Don Lewis
On 5 Jan, Denis Shaposhnikov wrote: >> "Don" == Don Lewis <[EMAIL PROTECTED]> writes: > > Don> pid 519 wants to lock this vnode but some other thread is > Don> holding the vnode lock. Unfortunately we don't know who the > Don> lock holder is because the message is truncated. > > Is it po

Re: Recurring problem: processes block accessing UFS file system

2006-01-05 Thread Denis Shaposhnikov
> "Don" == Don Lewis <[EMAIL PROTECTED]> writes: Don> pid 519 wants to lock this vnode but some other thread is Don> holding the vnode lock. Unfortunately we don't know who the Don> lock holder is because the message is truncated. Is it possible to find out the answer from the crashdump?

Re: Recurring problem: processes block accessing UFS file system

2006-01-05 Thread Don Lewis
On 5 Jan, Denis Shaposhnikov wrote: > Hi! > >> "Greg" == Greg Rivers <[EMAIL PROTECTED]> writes: > > Greg> It's taken more than a month, but the problem has recurred > Greg> without snapshots ever having been run. I've got a good trace > > I think that I have the same problem on a fresh

Re: Recurring problem: processes block accessing UFS file system

2006-01-05 Thread Denis Shaposhnikov
Hi! > "Greg" == Greg Rivers <[EMAIL PROTECTED]> writes: Greg> It's taken more than a month, but the problem has recurred Greg> without snapshots ever having been run. I've got a good trace I think that I have the same problem on a fresh CURRENT. For some processes I see MWCHAN = ufs and "

Re: Recurring problem: processes block accessing UFS file system

2006-01-04 Thread Greg Rivers
On Wed, 4 Jan 2006, Don Lewis wrote: How about "show buffer 0xdc76fe30"? db> show buffer 0xdc76fe30 buf at 0xdc76fe30 b_flags = 0x20a0 b_error = 0, b_bufsize = 16384, b_bcount = 16384, b_resid = 0 b_bufobj = (0xc8985610), b_data = 0xe1d6b000, b_blkno = 365086368 lockstatus = 0, excl count

Re: Recurring problem: processes block accessing UFS file system

2006-01-04 Thread Don Lewis
On 3 Jan, Greg Rivers wrote: > On Tue, 3 Jan 2006, Don Lewis wrote: >> Pid 87117 is playing with buf 0xdc76fe30 which is not locked, and is >> sleeping on the buf's b_xflags member. It looks like 87117 is waiting >> for an in-progress write to complete. There are a large number of other >> send

Re: Recurring problem: processes block accessing UFS file system

2006-01-03 Thread Greg Rivers
On Tue, 3 Jan 2006, Don Lewis wrote: db> show lockedbufs [snip] looks like this is the buf that pid 87150 is waiting for: buf at 0xdc713f50 b_flags = 0xa00200a0 b_error = 0, b_bufsize = 16384, b_bcount = 16384, b_resid = 0 b_bufobj = (0xc8985610), b_data = 0xe0b7b000, b_blkno = 365094624 lo

Re: Recurring problem: processes block accessing UFS file system

2006-01-03 Thread Don Lewis
On 3 Jan, Greg Rivers wrote: > On Tue, 3 Jan 2006, Don Lewis wrote: > >> There are large number of sendmail processes waiting on vnode locks >> which are held by other sendmail processes that are waiting on other >> vnode locks, etc. until we get to sendmail pid 87150 which is holding a >> vnode

Re: Recurring problem: processes block accessing UFS file system

2006-01-03 Thread Greg Rivers
On Tue, 3 Jan 2006, Don Lewis wrote: There are large number of sendmail processes waiting on vnode locks which are held by other sendmail processes that are waiting on other vnode locks, etc. until we get to sendmail pid 87150 which is holding a vnode lock and waiting to lock a buf. Tracing com

Re: Recurring problem: processes block accessing UFS file system

2006-01-03 Thread Don Lewis
On 3 Jan, Greg Rivers wrote: > On Tue, 22 Nov 2005, I wrote: > >> On Mon, 21 Nov 2005, Kris Kennaway wrote: >> >>> It may not be the same problem. You should also try to obtain a trace when >>> snapshots are not implicated. >>> >> >> Agreed. I'll do so at the first opportunity. >> > > First,

Re: Recurring problem: processes block accessing UFS file system

2006-01-03 Thread Greg Rivers
On Tue, 22 Nov 2005, I wrote: On Mon, 21 Nov 2005, Kris Kennaway wrote: It may not be the same problem. You should also try to obtain a trace when snapshots are not implicated. Agreed. I'll do so at the first opportunity. First, my thanks to all of you for looking into this. It's tak

Re: Recurring problem: processes block accessing UFS file system

2006-01-02 Thread Don Lewis
On 22 Nov, Greg Rivers wrote: > On Mon, 21 Nov 2005, Kris Kennaway wrote: > >> It may not be the same problem. You should also try to obtain a trace >> when snapshots are not implicated. >> > > Agreed. I'll do so at the first opportunity. > > >> 'show lockedvnods' is very important for diagn

Re: Recurring problem: processes block accessing UFS file system

2006-01-02 Thread Don Lewis
On 26 Nov, Tor Egge wrote: > >> Thanks Kris, these are exactly the clues I needed. Since the deadlock >> during a snapshot is fairly easy to reproduce, I did so and collected this >> information below. "alltrace" didn't work as I expected (didn't produce a >> trace), so I traced each pid asso

Re: Recurring problem: processes block accessing UFS file system

2006-01-02 Thread Don Lewis
On 26 Nov, Tor Egge wrote: > >> Thanks Kris, these are exactly the clues I needed. Since the deadlock >> during a snapshot is fairly easy to reproduce, I did so and collected this >> information below. "alltrace" didn't work as I expected (didn't produce a >> trace), so I traced each pid asso

Re: Recurring problem: processes block accessing UFS file system

2005-11-25 Thread Tor Egge
> Thanks Kris, these are exactly the clues I needed. Since the deadlock > during a snapshot is fairly easy to reproduce, I did so and collected this > information below. "alltrace" didn't work as I expected (didn't produce a > trace), so I traced each pid associated with a locked vnode separa

Re: Recurring problem: processes block accessing UFS file system

2005-11-24 Thread Peter Jeremy
On Mon, 2005-Nov-21 21:23:10 -0600, Greg Rivers wrote: > the >deadlock also occurs under normal operation when no snapshots are running >or have ever been run since boot. It's just much less frequent in this >case. I've also seen this behaviour in both 5.x and 6.0 (I can't recall if it bit me

Re: Recurring problem: processes block accessing UFS file system

2005-11-21 Thread Kris Kennaway
On Mon, Nov 21, 2005 at 09:23:10PM -0600, Greg Rivers wrote: > On Mon, 21 Nov 2005, Kris Kennaway wrote: > > >Looks like a UFS snapshot deadlock. Are you running something like dump > >-L on this filesystem, or making other use of snapshots? > > > > Indeed I am (dump -L), but as I said (not ver

Re: Recurring problem: processes block accessing UFS file system

2005-11-21 Thread Greg Rivers
On Mon, 21 Nov 2005, Kris Kennaway wrote: Looks like a UFS snapshot deadlock. Are you running something like dump -L on this filesystem, or making other use of snapshots? Indeed I am (dump -L), but as I said (not very clearly, sorry), the deadlock also occurs under normal operation when no

Re: Recurring problem: processes block accessing UFS file system

2005-11-21 Thread Ronald Klop
On Tue, 22 Nov 2005 00:54:09 +0100, Greg Rivers <[EMAIL PROTECTED]> wrote: I've recently put up three busy email relay hosts running 6.0-STABLE. Performance is excellent except for a nagging critical issue that keeps cropping up. /var/spool is its own file system mounted on a geom stripe

Re: Recurring problem: processes block accessing UFS file system

2005-11-21 Thread Kris Kennaway
On Mon, Nov 21, 2005 at 05:54:09PM -0600, Greg Rivers wrote: > I've recently put up three busy email relay hosts running 6.0-STABLE. > Performance is excellent except for a nagging critical issue that keeps > cropping up. > > /var/spool is its own file system mounted on a geom stripe of four BSD