On 06/20/10 23:15, Fabian Keil wrote:
Lawrence Stewart<lstew...@freebsd.org> wrote:
On 06/20/10 22:28, Fabian Keil wrote:
Lawrence Stewart<lstew...@freebsd.org> wrote:
On 06/20/10 21:15, Fabian Keil wrote:
Lawrence Stewart<lstew...@freebsd.org> wrote:
On 06/20/10 03:58, Fabian Keil wrote:
Lawrence Stewart<lstew...@freebsd.org> wrote:
On 06/13/10 18:12, Lawrence Stewart wrote:
The time has come to solicit some external testing for my SIFTR tool.
I'm hoping to commit it within a week or so unless problems are discovered.
I'm interested in all feedback and reports of success/failure, along
with details of the architecture tested and number of CPUs if you would
be so kind.
I got the following hand-transcribed panic maybe a second after
sysctl net.inet.siftr.enabled=1
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
[...]
current process = 12 (swi4: clock)
[ thread pid 12 tid 100006 ]
Stopped at siftr_chkpkt+0xd0: addq $0x1,0x8(%r14)
db> where
Tracing pid 12 tid 100006 td 0xffffff00034037e0
siftr_chkpt() at siftr_chkpkt+0xd0
pfil_run_hooks() at pfil_run_hooks+0xb4
ip_output() at ip_output+0x382
tcp_output() tcp_output+0xa41
tcp_timer_rexmt() at tcp_timer_rexmt+0x251
softclock() at softclock+0x291
intr_event_execute_handlers() at intr_event_execute_handlers+0x66
ithread_loop at ithread_loop+0x8e
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800003ad30, rbp = 0 ---
So I've tracked down the line of code where the page fault is occurring:
if (dir == PFIL_IN)
ss->n_in++;
else
ss->n_out++;
ss is a DPCPU (dynamic per-cpu) variable used to keep a set of stats
per-cpu and is initialised at the start of the function like so:
ss = DPCPU_PTR(ss);
So for ss to be NULL, that implies DPCPU_PTR() is returning NULL on your
machine. I know very little about the inner workings of the DPCPU_*
macros, but I'm pretty sure the way I use them in SIFTR is correct or at
least as intended.
siftr_chkpkt() passes ss to siftr_chkreinject() before dereferencing
it itself. I think if ss was NULL, the panic should already occur in
siftr_chkreinject().
Yes but siftr_chkreinject() only dereferences ss in the exceptional case
of a malloc failure or duplicate pkt. It's unlikely either case happens
for you and so wouldn't trigger the panic.
To be sure I added:
diff --git a/sys/netinet/siftr.c b/sys/netinet/siftr.c
index 8bc3498..b9fdfe4 100644
--- a/sys/netinet/siftr.c
+++ b/sys/netinet/siftr.c
@@ -788,6 +788,16 @@ siftr_chkpkt(void *arg, struct mbuf **m, struct ifnet
*ifp, int dir,
if (siftr_chkreinject(*m, dir, ss))
goto ret;
+ if (ss == NULL) {
+ printf("ss is NULL");
+ ss = DPCPU_PTR(ss);
+ if (ss == NULL) {
+ printf("ss is still NULL");
+ goto ret;
+ }
+ }
+
+
if (dir == PFIL_IN)
ss->n_in++;
else
which doesn't seem to affect the problem.
As in it still panics and the "ss is NULL" message is not printed? I
would have expected to at least see "ss is NULL" printed if my
hypothesis was correct... hmm.
Yes, it still panics, but no message is printed.
It was just pointed out to me that ss doesn't have to be NULL in order
to cause the page fault (duh). It could also just be a garbage ptr which
is why your print statement isn't firing.
Can you trigger the panic again and look for some information along the
lines of "fault virtual address = ..." as part of the panic info.
Knowing the faulting address would be useful and may help further diagnosis.
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0xffffff7f808f9de8
fault code = supervisor write data, page not present
instruction pointer = 0x20:0xffffffff8241f800
stack pointer = 0x28:0xffffff800003a7d0
frame pointer = 0x28:0xffffff800003a840
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
None of this looks too crazy, but at least one person I've been chatting
to about this thinks the faulting address doesn't look quite right for a
DPCPU variable.
Can you please get the following additional info from DDB:
show reg
show dpcpu_offset
p/x pcpu_entry_modspace
And can you also please identify the upstream FreeBSD revision number
your kernel source is based on (as opposed to the GIT rev) so we can
make sure we're looking at the same base sources you're running.
current process = 12 (swi4: clock)
[ thread pid 12 tid 100006 ]
Stopped at siftr_chkpkt+0xd0: addq $0x1,0x8(%r14)
db> where
Tracing pid 12 tid 100006 td 0xffffff00034037e0
siftr_chkpt() at siftr_chkpkt+0xd0
pfil_run_hooks() at pfil_run_hooks+0xb4
ip_output() at ip_output+0x382
tcp_output() tcp_output+0xa41
tcp_timer_rexmt() at tcp_timer_rexmt+0x251
softclock() at softclock+0x291
intr_event_execute_handlers() at intr_event_execute_handlers+0x66
ithread_loop at ithread_loop+0x8e
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800003ad30, rbp = 0 ---
Could you please go ahead and retest using a GENERIC kernel and see if
you can reproduce? There could be something in your custom kernel
causing the offsets or linker set magic used by the DPCPU bits to break
which in turn is triggering this panic in SIFTR.
I'll retry without pf first, and with GENERIC afterwards.
Sounds good, thanks.
Taking pf (and altq) out of the picture doesn't seem to make
a difference.
Wouldn't have expected it to. Will be very curious to know if the panic
is triggered in GENERIC.
It's not. I, too, get pfil.c related LORs though:
Ok good to know. Can you please share your kernel config that panics and
the modules you have loaded at the time of the panic? If we could
whittle your config down to something that's close to GENERIC and still
panics, that would obviously be useful to eliminate some variables.
The pfil related LORs are fine and can be ignored in this case.
lock order reversal:
1st 0xffffffff80e5c568 PFil hook read/write mutex (PFil hook read/write
mutex) @ /usr/src/sys/net/pfil.c:77
2nd 0xffffffff80e5dd68 udp (udp) @
/usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:3035
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_rw_rlock() at _rw_rlock+0x5f
pf_socket_lookup() at pf_socket_lookup+0x1c5
pf_test_udp() at pf_test_udp+0x8b0
pf_test() at pf_test+0x1089
pf_check_in() at pf_check_in+0x39
pfil_run_hooks() at pfil_run_hooks+0xcf
ip_input() at ip_input+0x2ae
swi_net() at swi_net+0x151
intr_event_execute_handlers() at intr_event_execute_handlers+0x66
ithread_loop() at ithread_loop+0xb2
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000044d30, rbp = 0 ---
lock order reversal:
1st 0xffffffff80e5c568 PFil hook read/write mutex (PFil hook read/write
mutex) @ /usr/src/sys/net/pfil.c:77
2nd 0xffffffff80e5d788 tcp (tcp) @
/usr/src/sys/modules/siftr/../../netinet/siftr.c:698
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x81e
_rw_rlock() at _rw_rlock+0x5f
siftr_chkpkt() at siftr_chkpkt+0x3c4
pfil_run_hooks() at pfil_run_hooks+0xcf
ip_input() at ip_input+0x2ae
swi_net() at swi_net+0x151
intr_event_execute_handlers() at intr_event_execute_handlers+0x66
ithread_loop() at ithread_loop+0xb2
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000044d30, rbp = 0 ---
My custom kernel normally doesn't have INVARIANTS and WITNESS
enabled, so I'll try to enable them next.
INVARIANTS and INVARIANT_SUPPORT are the import ones. I don't think
WITNESS will help us diagnose this particular problem.
Cheers,
Lawrence
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
