Re: nfsd oops on Linus' current tree.

2013-01-04 Thread Adamson, Dros
On Jan 3, 2013, at 6:26 PM, "Myklebust, Trond" wrote: > On Thu, 2013-01-03 at 18:11 -0500, Trond Myklebust wrote: >> On Thu, 2013-01-03 at 17:26 -0500, Tejun Heo wrote: >>> Ooh, BTW, there was a bug where workqueue code created a false >>> dependency between two work items. Workqueue currently

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread Myklebust, Trond
On Thu, 2013-01-03 at 18:11 -0500, Trond Myklebust wrote: > On Thu, 2013-01-03 at 17:26 -0500, Tejun Heo wrote: > > Ooh, BTW, there was a bug where workqueue code created a false > > dependency between two work items. Workqueue currently considers two > > work items to be the same if they're on th

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread Myklebust, Trond
On Thu, 2013-01-03 at 17:26 -0500, Tejun Heo wrote: > Hello, Trond. > > On Thu, Jan 03, 2013 at 10:12:32PM +, Myklebust, Trond wrote: > > > The analysis is likely completely wrong, so please don't go off doing > > > something unnecessary. Please take look at what's causing the > > > deadlocks

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread J. Bruce Fields
On Thu, Jan 03, 2013 at 05:03:09PM -0500, Tejun Heo wrote: > Hello, guys. > > On Thu, Jan 03, 2013 at 04:28:37PM +, Adamson, Dros wrote: > > The deadlock we were seeing was: > > > > - task A gets queued on rpciod workqueue and assigned kworker-0:0 > > - task B gets queued on rpciod workqueue

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread Tejun Heo
On Thu, Jan 03, 2013 at 05:26:39PM -0500, Tejun Heo wrote: > Hello, Trond. > > On Thu, Jan 03, 2013 at 10:12:32PM +, Myklebust, Trond wrote: > > > The analysis is likely completely wrong, so please don't go off doing > > > something unnecessary. Please take look at what's causing the > > > de

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread Tejun Heo
Hello, Trond. On Thu, Jan 03, 2013 at 10:12:32PM +, Myklebust, Trond wrote: > > The analysis is likely completely wrong, so please don't go off doing > > something unnecessary. Please take look at what's causing the > > deadlocks again. > > The analysis is a no-brainer: > We see a deadlock d

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread Tejun Heo
It's getting a bit repetitive but I really wanna steer people away from implementing separate kthreads for wrong reasons. kthread is surprisingly difficult to get right especially around freezing / exiting / hotplugs and people get it subtly wrong very often. On Thu, Jan 03, 2013 at 03:52:21PM -0

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread Myklebust, Trond
On Thu, 2013-01-03 at 17:08 -0500, Tejun Heo wrote: > Hello, > > On Thu, Jan 03, 2013 at 03:11:20PM -0500, J. Bruce Fields wrote: > > Both rpciod and nfsiod already set WQ_MEM_RECLAIM. > > > > But, right, looking at kernel/workqueue.c, it seems that the dedicated > > "rescuer" threads are invoked

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread Tejun Heo
Hello, On Thu, Jan 03, 2013 at 03:11:20PM -0500, J. Bruce Fields wrote: > Both rpciod and nfsiod already set WQ_MEM_RECLAIM. > > But, right, looking at kernel/workqueue.c, it seems that the dedicated > "rescuer" threads are invoked only in the case when work is stalled > because a new worker thre

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread Tejun Heo
Hello, guys. On Thu, Jan 03, 2013 at 04:28:37PM +, Adamson, Dros wrote: > The deadlock we were seeing was: > > - task A gets queued on rpciod workqueue and assigned kworker-0:0 > - task B gets queued on rpciod workqueue and assigned the same kworker > (kworker-0:0) > - task A gets run, calls

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread J. Bruce Fields
On Thu, Jan 03, 2013 at 08:27:20PM +, Adamson, Dros wrote: > > On Jan 3, 2013, at 3:11 PM, J. Bruce Fields wrote: > > > On Thu, Jan 03, 2013 at 04:28:37PM +, Adamson, Dros wrote: > >> Hey, sorry for the late response, I've been on vacation. > >> > >> On Dec 21, 2012, at 6:45 PM, J. Bruc

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread Adamson, Dros
On Jan 3, 2013, at 3:11 PM, J. Bruce Fields wrote: > On Thu, Jan 03, 2013 at 04:28:37PM +, Adamson, Dros wrote: >> Hey, sorry for the late response, I've been on vacation. >> >> On Dec 21, 2012, at 6:45 PM, J. Bruce Fields >> wrote: >> >>> On Fri, Dec 21, 2012 at 11:36:51PM +, Myklebu

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread J. Bruce Fields
On Thu, Jan 03, 2013 at 04:28:37PM +, Adamson, Dros wrote: > Hey, sorry for the late response, I've been on vacation. > > On Dec 21, 2012, at 6:45 PM, J. Bruce Fields > wrote: > > > On Fri, Dec 21, 2012 at 11:36:51PM +, Myklebust, Trond wrote: > >> Please reread what I said. There was no

Re: nfsd oops on Linus' current tree.

2013-01-03 Thread Adamson, Dros
Hey, sorry for the late response, I've been on vacation. On Dec 21, 2012, at 6:45 PM, J. Bruce Fields wrote: > On Fri, Dec 21, 2012 at 11:36:51PM +, Myklebust, Trond wrote: >> Please reread what I said. There was no obvious circular dependency, >> because nfsiod and rpciod are separate workq

Re: nfsd oops on Linus' current tree.

2012-12-21 Thread J. Bruce Fields
On Fri, Dec 21, 2012 at 11:36:51PM +, Myklebust, Trond wrote: > Please reread what I said. There was no obvious circular dependency, > because nfsiod and rpciod are separate workqueues, both created with > WQ_MEM_RECLAIM. Oh, sorry, I read "rpciod" as "nfsiod"! > Dros' experience shows, howev

RE: nfsd oops on Linus' current tree.

2012-12-21 Thread Myklebust, Trond
assuming the former. From: J. Bruce Fields [bfie...@fieldses.org] Sent: Friday, December 21, 2012 6:26 PM To: Myklebust, Trond Cc: Dave Jones; Linux Kernel; linux-...@vger.kernel.org; Adamson, Dros Subject: Re: nfsd oops on Linus' current tree. On Fri, Dec 21, 2012 at 11:15:40P

Re: nfsd oops on Linus' current tree.

2012-12-21 Thread J. Bruce Fields
On Fri, Dec 21, 2012 at 11:15:40PM +, Myklebust, Trond wrote: > Apologies for top-posting. The SSD on my laptop died, and so I'm stuck using > webmail for this account... Fun! If that happens to me on this trip, I've got a week trying to hack the kernel from my cell phone > Our experien

RE: nfsd oops on Linus' current tree.

2012-12-21 Thread Myklebust, Trond
mber 21, 2012 6:08 PM To: Myklebust, Trond Cc: Dave Jones; Linux Kernel; linux-...@vger.kernel.org; Adamson, Dros Subject: Re: nfsd oops on Linus' current tree. On Fri, Dec 21, 2012 at 06:40:54PM +, Myklebust, Trond wrote: > On Fri, 2012-12-21 at 13:08 -0500, J. Bruce Fields wrote: >

Re: nfsd oops on Linus' current tree.

2012-12-21 Thread J. Bruce Fields
On Fri, Dec 21, 2012 at 06:40:54PM +, Myklebust, Trond wrote: > On Fri, 2012-12-21 at 13:08 -0500, J. Bruce Fields wrote: > > On Fri, Dec 21, 2012 at 10:33:48AM -0500, Dave Jones wrote: > > > Did a mount from a client (also running Linus current), and the > > > server spat this out.. > > > > >

Re: nfsd oops on Linus' current tree.

2012-12-21 Thread Myklebust, Trond
On Fri, 2012-12-21 at 13:08 -0500, J. Bruce Fields wrote: > On Fri, Dec 21, 2012 at 10:33:48AM -0500, Dave Jones wrote: > > Did a mount from a client (also running Linus current), and the > > server spat this out.. > > > > [ 6936.306135] [ cut here ] > > [ 6936.306154] WARN

Re: nfsd oops on Linus' current tree.

2012-12-21 Thread J. Bruce Fields
On Fri, Dec 21, 2012 at 10:33:48AM -0500, Dave Jones wrote: > Did a mount from a client (also running Linus current), and the > server spat this out.. > > [ 6936.306135] [ cut here ] > [ 6936.306154] WARNING: at net/sunrpc/clnt.c:617 > rpc_shutdown_client+0x12a/0x1b0 [sunr