Re: Found it! (was Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds())

2013-12-02 Thread Al Viro
On Mon, Dec 02, 2013 at 06:58:57PM -0800, Linus Torvalds wrote: > In other words, it's unsafe to protect reference counts inside objects > with anything but spinlocks and/or atomic refcounts. Or you have to > have the lock *outside* the object you're protecting (which is often > what you want for

Re: Found it! (was Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds())

2013-12-02 Thread Linus Torvalds
Once more on this issue, because I've been unable to stop thinking about it, and noticed that it's actually even subtler than I thought initially. On Mon, Dec 2, 2013 at 8:00 AM, Linus Torvalds wrote: > > All our reference counting etc seems right, but we have one very > subtle bug: on the freein

Re: Found it! (was Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds())

2013-12-02 Thread Al Viro
On Mon, Dec 02, 2013 at 08:00:38AM -0800, Linus Torvalds wrote: > On Sat, Nov 30, 2013 at 1:08 PM, Linus Torvalds > wrote: > > > > I still don't see what could be wrong with the pipe_inode_info thing, > > but the fact that it's been so consistent in your traces does make me > > suspect it really i

Re: Found it! (was Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds())

2013-12-02 Thread Ingo Molnar
* Al Viro wrote: > On Mon, Dec 02, 2013 at 05:27:55PM +0100, Ingo Molnar wrote: > > > It's not like there should be many (any?) VFS operations where a pipe > > is used via i_mutex and pipe->mutex in parallel, which would improve > > scalability - so I don't see the scalability advantage. (But

Re: Found it! (was Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds())

2013-12-02 Thread Al Viro
On Mon, Dec 02, 2013 at 05:27:55PM +0100, Ingo Molnar wrote: > It's not like there should be many (any?) VFS operations where a pipe > is used via i_mutex and pipe->mutex in parallel, which would improve > scalability - so I don't see the scalability advantage. (But I might > be missing somethi

Re: Found it! (was Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds())

2013-12-02 Thread Ingo Molnar
* Linus Torvalds wrote: > On Sat, Nov 30, 2013 at 1:08 PM, Linus Torvalds > wrote: > > > > I still don't see what could be wrong with the pipe_inode_info thing, > > but the fact that it's been so consistent in your traces does make me > > suspect it really is *that* particular slab. > > I thin

Found it! (was Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds())

2013-12-02 Thread Linus Torvalds
On Sat, Nov 30, 2013 at 1:08 PM, Linus Torvalds wrote: > > I still don't see what could be wrong with the pipe_inode_info thing, > but the fact that it's been so consistent in your traces does make me > suspect it really is *that* particular slab. I think I finally found it. I've spent waaayy to

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-11-30 Thread Linus Torvalds
On Sat, Nov 30, 2013 at 1:43 AM, Simon Kirby wrote: > > I turned on kmalloc-192 tracing to find what else is using it: struct > nfs_fh, struct bio, and struct cred. Poking around those, struct bio has > bi_cnt, but it is way down in the struct. struct cred has "usage", but it > comes first. You c

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-11-30 Thread Simon Kirby
On Sat, Nov 30, 2013 at 09:25:33AM -0800, Linus Torvalds wrote: > On Sat, Nov 30, 2013 at 1:43 AM, Simon Kirby wrote: > > > I turned on kmalloc-192 tracing to find what else is using it: struct > > nfs_fh, struct bio, and struct cred. Poking around those, struct bio has > > bi_cnt, but it is way

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-11-30 Thread Linus Torvalds
On Sat, Nov 30, 2013 at 1:43 AM, Simon Kirby wrote: > On Tue, Nov 26, 2013 at 03:16:09PM -0800, Linus Torvalds wrote: >> >> The pipe-info structure isn't using its own slab cache, it's just >> using "kmalloc()". So it by definition will merge with all other >> kmalloc() allocations of the same siz

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-11-30 Thread Simon Kirby
On Tue, Nov 26, 2013 at 03:16:09PM -0800, Linus Torvalds wrote: > On Mon, Nov 25, 2013 at 4:44 PM, Simon Kirby wrote: > > > > I was hoping this or something else by 3.12 would have fixed it, so after > > testing we deployed this everywhere and turned off the rest of the debug > > options. I misse

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-11-26 Thread Linus Torvalds
On Tue, Nov 26, 2013 at 3:16 PM, Linus Torvalds wrote: > > I'm really not very happy with the whole pipe locking logic (or the > refcounting we do, separately from the "struct inode"), and in that > sense I'm perfectly willing to blame that code for doing bad things. > But the fact that it all goe

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-11-26 Thread Linus Torvalds
On Mon, Nov 25, 2013 at 4:44 PM, Simon Kirby wrote: > > I was hoping this or something else by 3.12 would have fixed it, so after > testing we deployed this everywhere and turned off the rest of the debug > options. I missed slub_debug on one server, though...and it just hit > another case of over

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-11-25 Thread Simon Kirby
On Tue, Aug 20, 2013 at 12:51:11AM -0700, Ian Applegate wrote: > Unfortunately no boxen with CONFIG_DEBUG_MUTEXES among them. I can > enable on a few and should have some results within the day. These > mainly serve (quite a bit of) HTTP/S cache traffic. > > On Tue, Aug 20, 2013 at 12:21 AM, Al V

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-09-03 Thread Simon Kirby
On Mon, Aug 19, 2013 at 04:31:38PM -0700, Simon Kirby wrote: > On Mon, Aug 19, 2013 at 05:24:41PM -0400, Chris Mason wrote: > > > Quoting Linus Torvalds (2013-08-19 17:16:36) > > > On Mon, Aug 19, 2013 at 1:29 PM, Christoph Lameter > > > wrote: > > > > On Mon, 19 Aug 2013, Simon Kirby wrote: >

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-08-20 Thread Ian Applegate
Unfortunately no boxen with CONFIG_DEBUG_MUTEXES among them. I can enable on a few and should have some results within the day. These mainly serve (quite a bit of) HTTP/S cache traffic. On Tue, Aug 20, 2013 at 12:21 AM, Al Viro wrote: > On Tue, Aug 20, 2013 at 12:17:52AM -0700, Ian Applegate wrot

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-08-20 Thread Al Viro
On Tue, Aug 20, 2013 at 12:17:52AM -0700, Ian Applegate wrote: > We are also seeing this or a similar issue. On a fairly widespread > deployment of 3.10.1 & 3.10.6 this occurred fairly consistently on the > order of 36 days (combined MTBF.) Do you have any boxen with CONFIG_DEBUG_MUTEXES among tho

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-08-20 Thread Ian Applegate
We are also seeing this or a similar issue. On a fairly widespread deployment of 3.10.1 & 3.10.6 this occurred fairly consistently on the order of 36 days (combined MTBF.) [28974.739774] [ cut here ] [28974.744980] kernel BUG at mm/slub.c:3352! [28974.749502] invalid opcode

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-08-19 Thread Al Viro
On Mon, Aug 19, 2013 at 02:16:36PM -0700, Linus Torvalds wrote: > On Mon, Aug 19, 2013 at 1:29 PM, Christoph Lameter wrote: > > On Mon, 19 Aug 2013, Simon Kirby wrote: > > > >>[... ] The > >> alloc/free traces are always the same -- always alloc_pipe_info and > >> free_pipe_info. This is seen

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-08-19 Thread Simon Kirby
On Mon, Aug 19, 2013 at 05:24:41PM -0400, Chris Mason wrote: > Quoting Linus Torvalds (2013-08-19 17:16:36) > > On Mon, Aug 19, 2013 at 1:29 PM, Christoph Lameter wrote: > > > On Mon, 19 Aug 2013, Simon Kirby wrote: > > > > > >>[... ] The > > >> alloc/free traces are always the same -- alway

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-08-19 Thread Chris Mason
Quoting Linus Torvalds (2013-08-19 17:16:36) > On Mon, Aug 19, 2013 at 1:29 PM, Christoph Lameter wrote: > > On Mon, 19 Aug 2013, Simon Kirby wrote: > > > >>[... ] The > >> alloc/free traces are always the same -- always alloc_pipe_info and > >> free_pipe_info. This is seen on 3.10 and (now)

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-08-19 Thread Linus Torvalds
On Mon, Aug 19, 2013 at 1:29 PM, Christoph Lameter wrote: > On Mon, 19 Aug 2013, Simon Kirby wrote: > >>[... ] The >> alloc/free traces are always the same -- always alloc_pipe_info and >> free_pipe_info. This is seen on 3.10 and (now) 3.11-rc4: >> >> Object 880090f19e78: 6b 6b 6b 6b 6c 6

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-08-19 Thread Christoph Lameter
On Mon, 19 Aug 2013, Simon Kirby wrote: > Object 880090f19e78: 6b 6b 6b 6b 6c 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > lkkk This looks like an increment after free in the second 32 bit value of the structure. First 32 bit value's poison is unchanged. > CONFIG_EFENCE? :) I think the

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-08-19 Thread Simon Kirby
On Sat, Jul 06, 2013 at 11:27:38AM +0300, Pekka Enberg wrote: > On Sat, Jul 6, 2013 at 3:09 AM, Simon Kirby wrote: > > We saw two Oopses overnight on two separate boxes that seem possibly > > related, but both are weird. These boxes typically run btrfs for rsync > > snapshot backups (and usually

Re: [3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-07-06 Thread Pekka Enberg
On Sat, Jul 6, 2013 at 3:09 AM, Simon Kirby wrote: > We saw two Oopses overnight on two separate boxes that seem possibly > related, but both are weird. These boxes typically run btrfs for rsync > snapshot backups (and usually Oops in btrfs ;), but not this time! > backup02 was running 3.10-rc6 pl

[3.10] Oopses in kmem_cache_allocate() via prepare_creds()

2013-07-05 Thread Simon Kirby
We saw two Oopses overnight on two separate boxes that seem possibly related, but both are weird. These boxes typically run btrfs for rsync snapshot backups (and usually Oops in btrfs ;), but not this time! backup02 was running 3.10-rc6 plus btrfs-next at the time, and backup03 was running 3.10 rel