Re: OOPS in perf_mmap_close()

2013-06-05 Thread Peter Zijlstra
On Wed, Jun 05, 2013 at 11:54:52AM -0400, Vince Weaver wrote: > On Mon, 3 Jun 2013, Peter Zijlstra wrote: > > > Seems to cure it... I'm going to let it run over night. > > [sorry for the delay in testing, was at the hospital waiting for > a new baby, and the free wi-fi had the ssh port blocked]

Re: OOPS in perf_mmap_close()

2013-06-05 Thread Vince Weaver
On Mon, 3 Jun 2013, Peter Zijlstra wrote: > Seems to cure it... I'm going to let it run over night. [sorry for the delay in testing, was at the hospital waiting for a new baby, and the free wi-fi had the ssh port blocked] I'm running 3.10-rc4 with your patch plus the 3-line fixup applied and my

Re: OOPS in perf_mmap_close()

2013-06-03 Thread Peter Zijlstra
On Mon, Jun 03, 2013 at 03:26:45PM +0200, Peter Zijlstra wrote: > On Thu, May 30, 2013 at 08:51:00AM -0400, Vince Weaver wrote: > > > I'll go prod, thanks again! > > > OK the below builds and seems to survive both test cases and about 10 minutes > of fuzzing -- fingers crossed. > > --- > @@ -377

Re: OOPS in perf_mmap_close()

2013-06-03 Thread Peter Zijlstra
On Mon, Jun 03, 2013 at 03:26:45PM +0200, Peter Zijlstra wrote: > On Thu, May 30, 2013 at 08:51:00AM -0400, Vince Weaver wrote: > > > I'll go prod, thanks again! > > > OK the below builds and seems to survive both test cases and about 10 minutes > of fuzzing -- fingers crossed. OK, looks like I

Re: OOPS in perf_mmap_close()

2013-06-03 Thread Peter Zijlstra
On Thu, May 30, 2013 at 08:51:00AM -0400, Vince Weaver wrote: > > I'll go prod, thanks again! OK the below builds and seems to survive both test cases and about 10 minutes of fuzzing -- fingers crossed. --- include/linux/perf_event.h |3 +- kernel/events/core.c | 230

Re: OOPS in perf_mmap_close()

2013-05-31 Thread Peter Zijlstra
On Thu, May 30, 2013 at 08:51:00AM -0400, Vince Weaver wrote: > On Thu, 30 May 2013, Peter Zijlstra wrote: > > I'll go prod, thanks again! > > The bug looks related to hw breakpoints, not sure if the bugs Oleg > Nesterov has been reporting in this area might be relevant or not. Nah its not. I fi

Re: OOPS in perf_mmap_close()

2013-05-30 Thread Vince Weaver
On Thu, 30 May 2013, Peter Zijlstra wrote: > > Awesome! How specific is it to perf? I mean, would that tool work > equally well for other tinity report? No, it's fairly specific to my perf_fuzzer. I only use trinity code to set up the perf_event_open() attr parameters, everything else I use my

Re: OOPS in perf_mmap_close()

2013-05-30 Thread Peter Zijlstra
On Wed, May 29, 2013 at 03:18:32PM -0400, Vince Weaver wrote: > On Wed, 29 May 2013, Peter Zijlstra wrote: > > > > Hurm.. I don't suppose you have an easy reproducer handy eh? I'll go > > stare at it. At least the current state is better than before, but > > clearly we're not quite there yet. > >

Re: OOPS in perf_mmap_close()

2013-05-29 Thread Vince Weaver
On Wed, 29 May 2013, Peter Zijlstra wrote: > > Hurm.. I don't suppose you have an easy reproducer handy eh? I'll go > stare at it. At least the current state is better than before, but > clearly we're not quite there yet. OK, below is an easy reproducer. Just run it two or three times. It leaks

Re: OOPS in perf_mmap_close()

2013-05-29 Thread Vince Weaver
On Wed, 29 May 2013, Peter Zijlstra wrote: > Hurm.. I don't suppose you have an easy reproducer handy eh? I'll go > stare at it. At least the current state is better than before, but > clearly we're not quite there yet. I've been working on a reproducible test case. Simple modifications to the

Re: OOPS in perf_mmap_close()

2013-05-29 Thread Ingo Molnar
* Vince Weaver wrote: > On Tue, 28 May 2013, Peter Zijlstra wrote: > > > Take 2.. :-) > > > > Leaving the locked vs pinned debacle for another time. The below patch > > survives many non-priv execution of the problem prog. > > It looks like this is already in tip, [...] Wanted to help out w

Re: OOPS in perf_mmap_close()

2013-05-29 Thread Peter Zijlstra
On Tue, May 28, 2013 at 02:22:11PM -0400, Vince Weaver wrote: > On Tue, 28 May 2013, Vince Weaver wrote: > > > It looks like this is already in tip, but I can confirm that this > > patch seems to fix things on my machine and holds up against longer > > fuzzing runs. > > OK, I take it back. Eve

Re: OOPS in perf_mmap_close()

2013-05-28 Thread Vince Weaver
On Tue, 28 May 2013, Vince Weaver wrote: > It looks like this is already in tip, but I can confirm that this > patch seems to fix things on my machine and holds up against longer > fuzzing runs. OK, I take it back. Even with the new patch applied, my fuzzer can still make the kernel leak user

Re: OOPS in perf_mmap_close()

2013-05-28 Thread Vince Weaver
On Tue, 28 May 2013, Peter Zijlstra wrote: > Take 2.. :-) > > Leaving the locked vs pinned debacle for another time. The below patch > survives many non-priv execution of the problem prog. It looks like this is already in tip, but I can confirm that this patch seems to fix things on my machine

Re: OOPS in perf_mmap_close()

2013-05-28 Thread Peter Zijlstra
Take 2.. :-) Leaving the locked vs pinned debacle for another time. The below patch survives many non-priv execution of the problem prog. --- Subject: perf: Fix perf mmap issues Vince reported a problem found by his perf specific trinity fuzzer. Al noticed 2 problems with perf's mmap(): - i

Re: OOPS in perf_mmap_close()

2013-05-24 Thread Peter Zijlstra
On Thu, May 23, 2013 at 07:40:12PM -0400, Vince Weaver wrote: > On Thu, 23 May 2013, Peter Zijlstra wrote: > > > On Thu, May 23, 2013 at 10:10:36AM -0400, Vince Weaver wrote: > > > > > > I can confirm your patch avoids the oops on my machine. > > > > > > It does lead to interesting behavior if I

Re: OOPS in perf_mmap_close()

2013-05-23 Thread Vince Weaver
On Thu, 23 May 2013, Peter Zijlstra wrote: > On Thu, May 23, 2013 at 10:10:36AM -0400, Vince Weaver wrote: > > > > I can confirm your patch avoids the oops on my machine. > > > > It does lead to interesting behavior if I run the sample program > > multiple times (with added printfs): > > > > vi

Re: OOPS in perf_mmap_close()

2013-05-23 Thread Peter Zijlstra
On Thu, May 23, 2013 at 05:59:10PM +, Christoph Lameter wrote: > On Thu, 23 May 2013, Peter Zijlstra wrote: > > > I know all that, and its completely irrelevant to the discussion. > > What you said in the rest of the email seems to indicate that you > still do not know that and I am repeating

Re: OOPS in perf_mmap_close()

2013-05-23 Thread Christoph Lameter
On Thu, 23 May 2013, Peter Zijlstra wrote: > I know all that, and its completely irrelevant to the discussion. What you said in the rest of the email seems to indicate that you still do not know that and I am repeating what I have said before here. > You now have double the amount of memory you

Re: OOPS in perf_mmap_close()

2013-05-23 Thread Peter Zijlstra
On Thu, May 23, 2013 at 04:12:14PM +, Christoph Lameter wrote: > On Thu, 23 May 2013, Peter Zijlstra wrote: > > > The patch completely fails to explain how RLIMIT_LOCKED is supposed to > > deal with pinned vs locked. Perf used to account its pages against > > RLIMIT_LOCKED, with the patch it c

Re: OOPS in perf_mmap_close()

2013-05-23 Thread Christoph Lameter
On Thu, 23 May 2013, Peter Zijlstra wrote: > The patch completely fails to explain how RLIMIT_LOCKED is supposed to > deal with pinned vs locked. Perf used to account its pages against > RLIMIT_LOCKED, with the patch it compares pinned against RLIMIT_LOCKED > but completely discards any possible l

Re: OOPS in perf_mmap_close()

2013-05-23 Thread Vince Weaver
On Thu, 23 May 2013, Peter Zijlstra wrote: > On Thu, May 23, 2013 at 10:10:36AM -0400, Vince Weaver wrote: > > > > I can confirm your patch avoids the oops on my machine. > > > > It does lead to interesting behavior if I run the sample program > > multiple times (with added printfs): > > > > vi

Re: OOPS in perf_mmap_close()

2013-05-23 Thread Peter Zijlstra
On Thu, May 23, 2013 at 10:10:36AM -0400, Vince Weaver wrote: > > I can confirm your patch avoids the oops on my machine. > > It does lead to interesting behavior if I run the sample program > multiple times (with added printfs): > > vince@core2:~$ ./perf_mmap_close_bug > mmap1=0x7f06a6e9 >

Re: OOPS in perf_mmap_close()

2013-05-23 Thread Peter Zijlstra
On Thu, May 23, 2013 at 02:09:10PM +, Christoph Lameter wrote: > On Thu, 23 May 2013, Peter Zijlstra wrote: > > > Right it doesn't. I think the easiest solution for now is to not copy the > > VMA > > on fork(). > > Right. Pinned pages are not inherited. If a page is unpinned then that is > g

Re: OOPS in perf_mmap_close()

2013-05-23 Thread Christoph Lameter
On Thu, 23 May 2013, Peter Zijlstra wrote: > Right it doesn't. I think the easiest solution for now is to not copy the VMA > on fork(). Right. Pinned pages are not inherited. If a page is unpinned then that is going to happen for all address spaces that reference the page. > But I totally missed

Re: OOPS in perf_mmap_close()

2013-05-23 Thread Vince Weaver
I can confirm your patch avoids the oops on my machine. It does lead to interesting behavior if I run the sample program multiple times (with added printfs): vince@core2:~$ ./perf_mmap_close_bug mmap1=0x7f06a6e9 mmap2=0x7f06a6e7f000 vince@core2:~$ ./perf_mmap_close_bug mmap1=0x7f878a138000

Re: OOPS in perf_mmap_close()

2013-05-23 Thread Peter Zijlstra
The below fixes things for me. --- Subject: perf: Fix perf mmap issues Vince reported a problem found by his perf specific trinity fuzzer. Al noticed perf's mmap() has issues against fork() -- since we use vma->vm_mm -- and double mmap() -- we leak an rb refcount. While staring at this code I a

Re: OOPS in perf_mmap_close()

2013-05-23 Thread Peter Zijlstra
On Thu, May 23, 2013 at 05:48:03AM +0100, Al Viro wrote: > On Wed, May 22, 2013 at 11:48:51PM -0400, Vince Weaver wrote: > > > > In case anyone cares, the Oops is happening here: > > > > 1a56: 48 c1 e8 0c shr$0xc,%rax > > 1a5a: 48 ff c0inc%r

Re: OOPS in perf_mmap_close()

2013-05-22 Thread Al Viro
On Wed, May 22, 2013 at 11:48:51PM -0400, Vince Weaver wrote: > > In case anyone cares, the Oops is happening here: > > 1a56: 48 c1 e8 0c shr$0xc,%rax > 1a5a: 48 ff c0inc%rax > > 1a5d: f0 48 29 45 60 lock sub %rax,0x60(%rbp)

Re: OOPS in perf_mmap_close()

2013-05-22 Thread Vince Weaver
In case anyone cares, the Oops is happening here: 1a56: 48 c1 e8 0c shr$0xc,%rax 1a5a: 48 ff c0inc%rax > 1a5d: f0 48 29 45 60 lock sub %rax,0x60(%rbp) 1a62: 49 8b 46 40 mov0x40(%r14),%rax Which map

Re: OOPS in perf_mmap_close()

2013-05-22 Thread Vince Weaver
> In any case while letting it run I got the following OOPS on > Linux 3.10-rc2 Included below is test code (based on the fuzzer output) that reliably OOPSes my core2 machine. It's a bit long, but I'm remote from the machine now so I can't refine it (as the code locked up the machine the las

OOPS in perf_mmap_close()

2013-05-22 Thread Vince Weaver
Hello I was fuzzing the kernel using my "perf_fuzzer" tool that builds on top of the trinity fuzzer. (You can get it as part of of my perf_event_test suite https://github.com/deater/perf_event_tests ) In any case while letting it run I got the following OOPS on Linux 3.10-rc2 [142450.070877] I