Re: panic: spin lock held too long (RELENG_8 from today)

2011-09-06 Thread Hiroki Sato
Attilio Rao wrote in : at> This should be enough for someone NFS-aware to look into it. at> at> Were you also able to get a core? Yes. But as kib@ pointed out it seems a deadlock in ZFS. Some experiments I did showed that this deadlock can be triggered at least by doing "rm -rf" against a

Re: panic: spin lock held too long (RELENG_8 from today)

2011-09-03 Thread Kostik Belousov
On Sat, Sep 03, 2011 at 12:05:47PM +0200, Attilio Rao wrote: > This should be enough for someone NFS-aware to look into it. > > Were you also able to get a core? > > I'll try to look into it in the next days, in particular about the > softclock state. > I am absolutely sure that this is a zfs d

Re: panic: spin lock held too long (RELENG_8 from today)

2011-09-01 Thread Attilio Rao
2011/9/1 Trent Nelson : > > On Aug 19, 2011, at 7:53 PM, Attilio Rao wrote: > >> If nobody complains about it earlier, I'll propose the patch to re@ in 8 >> hours. > > Just a friendly 'me too', for the records.  22 hours of heavy network/disk > I/O and no panic yet -- prior to the patch it was a

Re: panic: spin lock held too long (RELENG_8 from today)

2011-09-01 Thread Trent Nelson
On Aug 19, 2011, at 7:53 PM, Attilio Rao wrote: > If nobody complains about it earlier, I'll propose the patch to re@ in 8 > hours. Just a friendly 'me too', for the records. 22 hours of heavy network/disk I/O and no panic yet -- prior to the patch it was a panic orgy. Any response from re@

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-19 Thread Hiroki Sato
Attilio Rao wrote in : at> If nobody complains about it earlier, I'll propose the patch to re@ in 8 hours. Running fine for 45 hours so far. Please go ahead! -- Hiroki pgp3JVRs7kKa0.pgp Description: PGP signature

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-19 Thread Attilio Rao
If nobody complains about it earlier, I'll propose the patch to re@ in 8 hours. Attilio 2011/8/19 Mike Tancsa : > On 8/18/2011 8:37 PM, Chip Camden wrote: > >>> st> Thanks, Attilio.  I've applied the patch and removed the extra debug >>> st> options I had added (though keeping debug symbols).  I'

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-19 Thread Chip Camden
Quoth Mike Tancsa on Friday, 19 August 2011: > On 8/18/2011 8:37 PM, Chip Camden wrote: > > >> st> Thanks, Attilio. I've applied the patch and removed the extra debug > >> st> options I had added (though keeping debug symbols). I'll let you know > >> if > >> st> I experience any more panics. >

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-19 Thread Mike Tancsa
On 8/18/2011 8:37 PM, Chip Camden wrote: >> st> Thanks, Attilio. I've applied the patch and removed the extra debug >> st> options I had added (though keeping debug symbols). I'll let you know if >> st> I experience any more panics. >> >> No panic for 20 hours at this moment, FYI. For my NFS s

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-18 Thread Chip Camden
Quoth Hiroki Sato on Friday, 19 August 2011: > Chip Camden wrote > in <20110818025550.ga1...@libertas.local.camdensoftware.com>: > > st> Quoth Attilio Rao on Thursday, 18 August 2011: > st> > In callout_cpu_switch() if a low priority thread is migrating the > st> > callout and gets preempted af

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-18 Thread Hiroki Sato
Chip Camden wrote in <20110818025550.ga1...@libertas.local.camdensoftware.com>: st> Quoth Attilio Rao on Thursday, 18 August 2011: st> > In callout_cpu_switch() if a low priority thread is migrating the st> > callout and gets preempted after the outcoming cpu queue lock is left st> > (and sched

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-17 Thread Chip Camden
Quoth Attilio Rao on Thursday, 18 August 2011: > 2011/8/18 Hiroki Sato : > > Hiroki Sato wrote > >  in <20110818.043332.27079545013461535@allbsd.org>: > > > > hr> Attilio Rao wrote > > hr>   in > > : > > hr> > > hr> at> 2011/8/17 Hiroki Sato : > > hr> at> > Hi, > > hr> at> > > > hr> at> > Mi

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-17 Thread Jeremy Chadwick
On Wed, Aug 17, 2011 at 05:01:05PM -0700, Chip Camden wrote: > Quoth Jeremy Chadwick on Wednesday, 17 August 2011: > > > > > > I'm also getting similar panics on 8.2-STABLE. Locks up everything and I > > > have to power off. Once, I happened to be looking at the console when it > > > happened an

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-17 Thread Attilio Rao
2011/8/18 Hiroki Sato : > Hiroki Sato wrote >  in <20110818.043332.27079545013461535@allbsd.org>: > > hr> Attilio Rao wrote > hr>   in : > hr> > hr> at> 2011/8/17 Hiroki Sato : > hr> at> > Hi, > hr> at> > > hr> at> > Mike Tancsa wrote > hr> at> >  in <4e15a08c.6090...@sentex.net>: > hr> at>

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-17 Thread Attilio Rao
2011/8/18 Hiroki Sato : > Hiroki Sato wrote >  in <20110818.043332.27079545013461535@allbsd.org>: > > hr> Attilio Rao wrote > hr>   in : > hr> > hr> at> 2011/8/17 Hiroki Sato : > hr> at> > Hi, > hr> at> > > hr> at> > Mike Tancsa wrote > hr> at> >  in <4e15a08c.6090...@sentex.net>: > hr> at>

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-17 Thread Hiroki Sato
Hiroki Sato wrote in <20110818.043332.27079545013461535@allbsd.org>: hr> Attilio Rao wrote hr> in : hr> hr> at> 2011/8/17 Hiroki Sato : hr> at> > Hi, hr> at> > hr> at> > Mike Tancsa wrote hr> at> >  in <4e15a08c.6090...@sentex.net>: hr> at> > hr> at> > mi> On 7/7/2011 7:32 AM, Mike Tan

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-17 Thread Chip Camden
Quoth Jeremy Chadwick on Wednesday, 17 August 2011: > > > > I'm also getting similar panics on 8.2-STABLE. Locks up everything and I > > have to power off. Once, I happened to be looking at the console when it > > happened and copied dow the following: > > > > Sleeping thread (tif 100037, pid 0

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-17 Thread Jeremy Chadwick
On Wed, Aug 17, 2011 at 10:52:01AM -0700, Chip Camden wrote: > Quoth Hiroki Sato on Thursday, 18 August 2011: > > Hi, > > > > Mike Tancsa wrote > > in <4e15a08c.6090...@sentex.net>: > > > > mi> On 7/7/2011 7:32 AM, Mike Tancsa wrote: > > mi> > On 7/7/2011 4:20 AM, Kostik Belousov wrote: > > mi

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-17 Thread Hiroki Sato
Attilio Rao wrote in : at> 2011/8/17 Hiroki Sato : at> > Hi, at> > at> > Mike Tancsa wrote at> >  in <4e15a08c.6090...@sentex.net>: at> > at> > mi> On 7/7/2011 7:32 AM, Mike Tancsa wrote: at> > mi> > On 7/7/2011 4:20 AM, Kostik Belousov wrote: at> > mi> >> at> > mi> >> BTW, we had a similar pa

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-17 Thread Attilio Rao
2011/8/17 Hiroki Sato : > Hi, > > Mike Tancsa wrote >  in <4e15a08c.6090...@sentex.net>: > > mi> On 7/7/2011 7:32 AM, Mike Tancsa wrote: > mi> > On 7/7/2011 4:20 AM, Kostik Belousov wrote: > mi> >> > mi> >> BTW, we had a similar panic, "spinlock held too long", the spinlock > mi> >> is the sched l

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-17 Thread Mike Tancsa
On 8/17/2011 1:38 PM, Hiroki Sato wrote: > Any progress on the investigation? Unfortunately, I cannot reproduce it yet with a debugging kernel :( ---Mike > > -- > spin lock 0x80cb46c0 (sched lock 0) held by 0xff01900458c0 (tid > 100489) too long > panic: spin lock held to

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-17 Thread Chip Camden
Quoth Hiroki Sato on Thursday, 18 August 2011: > Hi, > > Mike Tancsa wrote > in <4e15a08c.6090...@sentex.net>: > > mi> On 7/7/2011 7:32 AM, Mike Tancsa wrote: > mi> > On 7/7/2011 4:20 AM, Kostik Belousov wrote: > mi> >> > mi> >> BTW, we had a similar panic, "spinlock held too long", the spinlo

Re: panic: spin lock held too long (RELENG_8 from today)

2011-08-17 Thread Hiroki Sato
Hi, Mike Tancsa wrote in <4e15a08c.6090...@sentex.net>: mi> On 7/7/2011 7:32 AM, Mike Tancsa wrote: mi> > On 7/7/2011 4:20 AM, Kostik Belousov wrote: mi> >> mi> >> BTW, we had a similar panic, "spinlock held too long", the spinlock mi> >> is the sched lock N, on busy 8-core box recently upgrad

Re: panic: spin lock held too long (RELENG_8 from today)

2011-07-07 Thread Mike Tancsa
On 7/7/2011 7:32 AM, Mike Tancsa wrote: > On 7/7/2011 4:20 AM, Kostik Belousov wrote: >> >> BTW, we had a similar panic, "spinlock held too long", the spinlock >> is the sched lock N, on busy 8-core box recently upgraded to the >> stable/8. Unfortunately, machine hung dumping core, so the stack tra

Re: panic: spin lock held too long (RELENG_8 from today)

2011-07-07 Thread Andriy Gapon
on 07/07/2011 14:41 Jeremy Chadwick said the following: > 1. info threads > 2. Find the index value that matches the tid in question (in the above >spin lock panic, that'd be tid 100109). The index value will be >the first number shown on the left > 3. thread {index} Just in case, in kgdb

Re: panic: spin lock held too long (RELENG_8 from today)

2011-07-07 Thread Jeremy Chadwick
On Thu, Jul 07, 2011 at 07:32:41AM -0400, Mike Tancsa wrote: > On 7/7/2011 4:20 AM, Kostik Belousov wrote: > > > > BTW, we had a similar panic, "spinlock held too long", the spinlock > > is the sched lock N, on busy 8-core box recently upgraded to the > > stable/8. Unfortunately, machine hung dump

Re: panic: spin lock held too long (RELENG_8 from today)

2011-07-07 Thread Mike Tancsa
On 7/7/2011 4:20 AM, Kostik Belousov wrote: > > BTW, we had a similar panic, "spinlock held too long", the spinlock > is the sched lock N, on busy 8-core box recently upgraded to the > stable/8. Unfortunately, machine hung dumping core, so the stack trace > for the owner thread was not available.

Re: panic: spin lock held too long (RELENG_8 from today)

2011-07-07 Thread Kostik Belousov
On Thu, Jul 07, 2011 at 10:36:42AM +0300, Andriy Gapon wrote: > on 07/07/2011 08:55 Mike Tancsa said the following: > > I did a buildworld on this box to bring it up to RELENG_8 for the BIND > > fixes. Unfortunately, the formerly solid box (April 13th kernel) > > panic'd tonight with > > > > Unre

Re: panic: spin lock held too long (RELENG_8 from today)

2011-07-07 Thread Andriy Gapon
on 07/07/2011 08:55 Mike Tancsa said the following: > I did a buildworld on this box to bring it up to RELENG_8 for the BIND > fixes. Unfortunately, the formerly solid box (April 13th kernel) > panic'd tonight with > > Unread portion of the kernel message buffer: > spin lock 0xc0b1d200 (sched loc