Re: Getting empty callchain from perf_callchain_kernel()

2019-06-13 Thread Josh Poimboeuf
On Wed, Jun 12, 2019 at 09:50:08AM -0500, Josh Poimboeuf wrote: > > Other than that, the same note as before, the 32bit JIT still seems > > buggered, but I'm not sure you (or anybody else) cares enough about that > > to fix it though. It seems to use ebp as its own frame pointer, which > > complete

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-12 Thread Josh Poimboeuf
On Wed, Jun 12, 2019 at 10:54:23AM +0200, Peter Zijlstra wrote: > On Tue, Jun 11, 2019 at 10:05:01PM -0500, Josh Poimboeuf wrote: > > On Fri, May 24, 2019 at 10:53:19AM +0200, Peter Zijlstra wrote: > > > > For ORC, I'm thinking we may be able to just require that all generated > > > > code (BPF and

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-12 Thread Josh Poimboeuf
On Wed, Jun 12, 2019 at 09:10:23AM -0400, Steven Rostedt wrote: > On Tue, 11 Jun 2019 22:05:01 -0500 > Josh Poimboeuf wrote: > > > Right now, ftrace has a special hook in the ORC unwinder > > (orc_ftrace_find). It would be great if we could get rid of that in > > favor of the "always use frame p

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-12 Thread Steven Rostedt
On Tue, 11 Jun 2019 22:05:01 -0500 Josh Poimboeuf wrote: > Right now, ftrace has a special hook in the ORC unwinder > (orc_ftrace_find). It would be great if we could get rid of that in > favor of the "always use frame pointers" approach. I'll hold off on > doing the kpatch/kprobe trampoline co

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-12 Thread Peter Zijlstra
On Tue, Jun 11, 2019 at 10:05:01PM -0500, Josh Poimboeuf wrote: > On Fri, May 24, 2019 at 10:53:19AM +0200, Peter Zijlstra wrote: > > > For ORC, I'm thinking we may be able to just require that all generated > > > code (BPF and others) always use frame pointers. Then when ORC doesn't > > > recogni

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-11 Thread Josh Poimboeuf
On Fri, May 24, 2019 at 10:53:19AM +0200, Peter Zijlstra wrote: > > For ORC, I'm thinking we may be able to just require that all generated > > code (BPF and others) always use frame pointers. Then when ORC doesn't > > recognize a code address, it could try using the frame pointer as a > > fallbac

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-11 Thread Josh Poimboeuf
On Thu, Jun 06, 2019 at 04:04:48PM +, Song Liu wrote: > Hi Josh, > > Have you got luck fixing the ORC side? Here's the ORC fix. It's needed in addition to the bpf frame pointer fix (the previous patch). I'll clean the patches up and post them soon. diff --git a/arch/x86/kernel/unwind_orc.

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-06 Thread Josh Poimboeuf
On Thu, Jun 06, 2019 at 04:04:48PM +, Song Liu wrote: > >> Here's a tentative BPF fix for the JIT frame pointer issue. It was a > >> bit harder than I expected. Encoding r12 as a base register requires a > >> SIB byte, so I had to add support for encoding that. I also simplified > >> the pro

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-06 Thread Song Liu
> On May 27, 2019, at 4:57 AM, Kairui Song wrote: > > On Sat, May 25, 2019 at 7:23 AM Josh Poimboeuf wrote: >> >> On Fri, May 24, 2019 at 10:20:52AM +0800, Kairui Song wrote: >>> On Fri, May 24, 2019 at 1:27 AM Josh Poimboeuf wrote: On Fri, May 24, 2019 at 12:41:59AM +0800, Kairu

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-27 Thread Kairui Song
On Sat, May 25, 2019 at 7:23 AM Josh Poimboeuf wrote: > > On Fri, May 24, 2019 at 10:20:52AM +0800, Kairui Song wrote: > > On Fri, May 24, 2019 at 1:27 AM Josh Poimboeuf wrote: > > > > > > On Fri, May 24, 2019 at 12:41:59AM +0800, Kairui Song wrote: > > > > On Thu, May 23, 2019 at 11:24 PM Josh

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-24 Thread Josh Poimboeuf
On Fri, May 24, 2019 at 10:20:52AM +0800, Kairui Song wrote: > On Fri, May 24, 2019 at 1:27 AM Josh Poimboeuf wrote: > > > > On Fri, May 24, 2019 at 12:41:59AM +0800, Kairui Song wrote: > > > On Thu, May 23, 2019 at 11:24 PM Josh Poimboeuf > > > wrote: > > > > > > > > On Thu, May 23, 2019 at 10

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-24 Thread Josh Poimboeuf
On Fri, May 24, 2019 at 10:53:19AM +0200, Peter Zijlstra wrote: > On Thu, May 23, 2019 at 10:24:13AM -0500, Josh Poimboeuf wrote: > > > Here's the latest version which should fix it in all cases (based on > > tip/master): > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/jpoimboe/lin

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-24 Thread Peter Zijlstra
On Thu, May 23, 2019 at 10:24:13AM -0500, Josh Poimboeuf wrote: > Here's the latest version which should fix it in all cases (based on > tip/master): > > > https://git.kernel.org/pub/scm/linux/kernel/git/jpoimboe/linux.git/commit/?h=bpf-orc-fix That patch suffers an inconsitency, the comment

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Kairui Song
On Fri, May 24, 2019 at 1:27 AM Josh Poimboeuf wrote: > > On Fri, May 24, 2019 at 12:41:59AM +0800, Kairui Song wrote: > > On Thu, May 23, 2019 at 11:24 PM Josh Poimboeuf > > wrote: > > > > > > On Thu, May 23, 2019 at 10:50:24PM +0800, Kairui Song wrote: > > > > > > Hi Josh, this still won't fi

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Josh Poimboeuf
On Fri, May 24, 2019 at 12:41:59AM +0800, Kairui Song wrote: > On Thu, May 23, 2019 at 11:24 PM Josh Poimboeuf wrote: > > > > On Thu, May 23, 2019 at 10:50:24PM +0800, Kairui Song wrote: > > > > > Hi Josh, this still won't fix the problem. > > > > > > > > > > Problem is not (or not only) with ___

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Kairui Song
On Thu, May 23, 2019 at 11:24 PM Josh Poimboeuf wrote: > > On Thu, May 23, 2019 at 10:50:24PM +0800, Kairui Song wrote: > > > > Hi Josh, this still won't fix the problem. > > > > > > > > Problem is not (or not only) with ___bpf_prog_run, what actually went > > > > wrong is with the JITed bpf code

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Josh Poimboeuf
On Thu, May 23, 2019 at 10:50:24PM +0800, Kairui Song wrote: > > > Hi Josh, this still won't fix the problem. > > > > > > Problem is not (or not only) with ___bpf_prog_run, what actually went > > > wrong is with the JITed bpf code. > > > > There seem to be a bunch of issues. My patch at least fixe

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Kairui Song
On Thu, May 23, 2019 at 9:32 PM Josh Poimboeuf wrote: > > On Thu, May 23, 2019 at 02:48:11PM +0800, Kairui Song wrote: > > On Thu, May 23, 2019 at 7:46 AM Josh Poimboeuf wrote: > > > > > > On Wed, May 22, 2019 at 12:45:17PM -0500, Josh Poimboeuf wrote: > > > > On Wed, May 22, 2019 at 02:49:07PM +

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Josh Poimboeuf
On Thu, May 23, 2019 at 02:48:11PM +0800, Kairui Song wrote: > On Thu, May 23, 2019 at 7:46 AM Josh Poimboeuf wrote: > > > > On Wed, May 22, 2019 at 12:45:17PM -0500, Josh Poimboeuf wrote: > > > On Wed, May 22, 2019 at 02:49:07PM +, Alexei Starovoitov wrote: > > > > The one that is broken is p

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Kairui Song
On Thu, May 23, 2019 at 4:28 PM Song Liu wrote: > > > On May 22, 2019, at 11:48 PM, Kairui Song wrote: > > > > On Thu, May 23, 2019 at 7:46 AM Josh Poimboeuf wrote: > >> > >> On Wed, May 22, 2019 at 12:45:17PM -0500, Josh Poimboeuf wrote: > >>> On Wed, May 22, 2019 at 02:49:07PM +, Alexei S

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Song Liu
> On May 22, 2019, at 11:48 PM, Kairui Song wrote: > > On Thu, May 23, 2019 at 7:46 AM Josh Poimboeuf wrote: >> >> On Wed, May 22, 2019 at 12:45:17PM -0500, Josh Poimboeuf wrote: >>> On Wed, May 22, 2019 at 02:49:07PM +, Alexei Starovoitov wrote: The one that is broken is prog_tests

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Kairui Song
On Thu, May 23, 2019 at 7:46 AM Josh Poimboeuf wrote: > > On Wed, May 22, 2019 at 12:45:17PM -0500, Josh Poimboeuf wrote: > > On Wed, May 22, 2019 at 02:49:07PM +, Alexei Starovoitov wrote: > > > The one that is broken is prog_tests/stacktrace_map.c > > > There we attach bpf to standard tracep

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Josh Poimboeuf
On Wed, May 22, 2019 at 12:45:17PM -0500, Josh Poimboeuf wrote: > On Wed, May 22, 2019 at 02:49:07PM +, Alexei Starovoitov wrote: > > The one that is broken is prog_tests/stacktrace_map.c > > There we attach bpf to standard tracepoint where > > kernel suppose to collect pt_regs before calling i

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Alexei Starovoitov
On 5/22/19 11:07 AM, Josh Poimboeuf wrote: > On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: >> On Fri, May 17, 2019 at 4:11 PM Peter Zijlstra wrote: >>> >>> On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote:

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Josh Poimboeuf
On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: > On Fri, May 17, 2019 at 4:11 PM Peter Zijlstra wrote: > > > > On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: > > > On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote: > > > > Hi, > > > > > > > > We found a failur

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Josh Poimboeuf
On Wed, May 22, 2019 at 02:49:07PM +, Alexei Starovoitov wrote: > The one that is broken is prog_tests/stacktrace_map.c > There we attach bpf to standard tracepoint where > kernel suppose to collect pt_regs before calling into bpf. > And that's what bpf_get_stackid_tp() is doing. > It passes pt

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Alexei Starovoitov
On 5/22/19 7:02 AM, Peter Zijlstra wrote: > >> If the unwinder could trace back through the bpf func call then there >> will be no such problem. > > Why couldn't it trace back through the bpf stuff? And how can we fix > that? Most of the time there is no 'tracing through bpf stuff'. bpf infra is

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Peter Zijlstra
On Mon, May 20, 2019 at 02:06:54AM +0800, Kairui Song wrote: > On Fri, May 17, 2019 at 5:10 PM Peter Zijlstra wrote: > > > > On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: > > > Hi, I think the actual problem is that bpf_get_stackid_tp (and maybe > > > some other bfp functions) is no

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Peter Zijlstra
On Mon, May 20, 2019 at 05:22:12PM +, Song Liu wrote: > I think this is still the best fix/workaround here? And only one level > of stack trace should be OK for tracepoint? No. That's still completely broken.

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-20 Thread Song Liu
> On May 19, 2019, at 11:07 AM, Kairui Song wrote: > > On Sat, May 18, 2019 at 5:48 AM Song Liu wrote: >> >> >> >>> On May 17, 2019, at 2:06 PM, Alexei Starovoitov wrote: >>> >>> On 5/17/19 11:40 AM, Song Liu wrote: +Alexei, Daniel, and bpf > On May 17, 2019, at 2:10 AM, P

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-20 Thread Song Liu
Sorry for previous empty email.. Clicked send by accident. > On May 19, 2019, at 11:06 AM, Kairui Song wrote: > > On Fri, May 17, 2019 at 5:10 PM Peter Zijlstra wrote: >> >> On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: >>> Hi, I think the actual problem is that bpf_get_stackid

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-20 Thread Song Liu
> On May 19, 2019, at 11:06 AM, Kairui Song wrote: > > On Fri, May 17, 2019 at 5:10 PM Peter Zijlstra wrote: >> >> On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: >>> Hi, I think the actual problem is that bpf_get_stackid_tp (and maybe >>> some other bfp functions) is now broken

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-19 Thread Kairui Song
On Sat, May 18, 2019 at 5:48 AM Song Liu wrote: > > > > > On May 17, 2019, at 2:06 PM, Alexei Starovoitov wrote: > > > > On 5/17/19 11:40 AM, Song Liu wrote: > >> +Alexei, Daniel, and bpf > >> > >>> On May 17, 2019, at 2:10 AM, Peter Zijlstra wrote: > >>> > >>> On Fri, May 17, 2019 at 04:15:39PM

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-19 Thread Kairui Song
On Fri, May 17, 2019 at 5:10 PM Peter Zijlstra wrote: > > On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: > > Hi, I think the actual problem is that bpf_get_stackid_tp (and maybe > > some other bfp functions) is now broken, or, strating an unwind > > directly inside a bpf program will

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Song Liu
> On May 17, 2019, at 2:06 PM, Alexei Starovoitov wrote: > > On 5/17/19 11:40 AM, Song Liu wrote: >> +Alexei, Daniel, and bpf >> >>> On May 17, 2019, at 2:10 AM, Peter Zijlstra wrote: >>> >>> On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: Hi, I think the actual problem is

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Alexei Starovoitov
On 5/17/19 11:40 AM, Song Liu wrote: > +Alexei, Daniel, and bpf > >> On May 17, 2019, at 2:10 AM, Peter Zijlstra wrote: >> >> On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: >>> Hi, I think the actual problem is that bpf_get_stackid_tp (and maybe >>> some other bfp functions) is now

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Song Liu
+Alexei, Daniel, and bpf > On May 17, 2019, at 2:10 AM, Peter Zijlstra wrote: > > On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: >> Hi, I think the actual problem is that bpf_get_stackid_tp (and maybe >> some other bfp functions) is now broken, or, strating an unwind >> directly in

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Song Liu
> On May 17, 2019, at 1:10 AM, Peter Zijlstra wrote: > > On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: >> On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote: >>> Hi, >>> >>> We found a failure with selftests/bpf/tests_prog in test_stacktrace_map (on >>> bpf/master >>

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Song Liu
> On May 17, 2019, at 1:32 AM, Kairui Song wrote: > > On Fri, May 17, 2019 at 4:15 PM Kairui Song wrote: >> >> On Fri, May 17, 2019 at 4:11 PM Peter Zijlstra wrote: >>> >>> On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: On Thu, May 16, 2019 at 11:51:55PM +, Song L

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Peter Zijlstra
On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: > Hi, I think the actual problem is that bpf_get_stackid_tp (and maybe > some other bfp functions) is now broken, or, strating an unwind > directly inside a bpf program will end up strangely. It have following > kernel message: Urgh, wha

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Kairui Song
On Fri, May 17, 2019 at 4:15 PM Kairui Song wrote: > > On Fri, May 17, 2019 at 4:11 PM Peter Zijlstra wrote: > > > > On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: > > > On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote: > > > > Hi, > > > > > > > > We found a failure with

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Kairui Song
On Fri, May 17, 2019 at 4:11 PM Peter Zijlstra wrote: > > On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: > > On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote: > > > Hi, > > > > > > We found a failure with selftests/bpf/tests_prog in test_stacktrace_map > > > (on bpf/maste

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Peter Zijlstra
On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: > On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote: > > Hi, > > > > We found a failure with selftests/bpf/tests_prog in test_stacktrace_map (on > > bpf/master > > branch). > > > > After digging into the code, we found that

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Peter Zijlstra
On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote: > Hi, > > We found a failure with selftests/bpf/tests_prog in test_stacktrace_map (on > bpf/master > branch). > > After digging into the code, we found that perf_callchain_kernel() is giving > empty > callchain for tracepoint sched/sch

Getting empty callchain from perf_callchain_kernel()

2019-05-16 Thread Song Liu
Hi, We found a failure with selftests/bpf/tests_prog in test_stacktrace_map (on bpf/master branch). After digging into the code, we found that perf_callchain_kernel() is giving empty callchain for tracepoint sched/sched_switch. And it seems related to commit d15d356887e770c5f2dcf963b52c7cb51