crash at in_pcb.c
Hello List,
I can realiably reproduce this crash. We have a deamon that accept several
connections
per sec. We use iperf and Microsoft Web application stress 1.0 to push
traffic to the FreeBSD box.
Without further delay, the crash dump is below. I've been troubleshooting,
but I am no longer sure
if this is a race condition or a stack corruption. The socket pointer
between frame 12 and 11 is different.
This is on 6.2, but the code for 7.0 is identical, so I think it still
applies.
Any hint, patching or troubleshooting this is appreciated.
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x2aef0210
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc0769098
stack pointer = 0x28:0xef781bc0
frame pointer = 0x28:0xef781bd0
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 1166 (ndaemon)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 8h32m25s
Dumping 3325 MB (3 chunks)
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) l *0xc0769098
0xc0769098 is in in_pcblookup_local (/usr/src/sys/netinet/in_pcb.c:923).
918 /usr/src/sys/netinet/in_pcb.c: No such file or directory.
in /usr/src/sys/netinet/in_pcb.c
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc06c2812 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:412
#2 0xc06c2bbd in panic (fmt=0xc0940872 "%s") at
/usr/src/sys/kern/kern_shutdown.c:573
#3 0xc08f3e4e in trap_fatal (frame=0xef781b80, eva=720306704) at
/usr/src/sys/i386/i386/trap.c:838
#4 0xc08f3b57 in trap_pfault (frame=0xef781b80, usermode=0, eva=720306704)
at /usr/src/sys/i386/i386/trap.c:745
#5 0xc08f3745 in trap (frame=
{tf_fs = -277348344, tf_es = 40, tf_ds = -913309656, tf_edi = 6,
tf_esi = 0, tf_ebp = -277341232, tf_isp = -277341268, tf_ebx = -1062683820,
tf_edx = 720306704, tf_ecx = 14063, tf_eax = 720306704, tf_trapno = 12,
tf_err = 0, tf_eip = -1065971560, tf_cs = 32, tf_eflags = 66050, tf_esp = 0,
tf_ss = -1062683820}) at /usr/src/sys/i386/i386/trap.c:435
#6 0xc08dddba in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr =
0}, lport_arg=720306704, wild_okay=1)
at /usr/src/sys/netinet/in_pcb.c:923
#8 0xc0768452 in in_pcbbind_setup (inp=0xc97150b4, nam=0x36ef,
laddrp=0xc97150ec, lportp=0xc97150ce, cred=0xc8726780)
at /usr/src/sys/netinet/in_pcb.c:464
#9 0xc0767f56 in in_pcbbind (inp=0xc97150b4, nam=0x2aef0210,
cred=0xc8726780) at /usr/src/sys/netinet/in_pcb.c:240
#10 0xc077f272 in tcp_connect (tp=0xc9897000, nam=0xc98a1ba0, td=0xc990e180)
at /usr/src/sys/netinet/tcp_usrreq.c:864
#11 0xc077e141 in tcp_usr_connect (so=0xc9897000, nam=0xc98a1ba0,
td=0xc990e180)
at /usr/src/sys/netinet/tcp_usrreq.c:369
#12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180)
at /usr/src/sys/kern/uipc_socket.c:558
#13 0xc07046a8 in kern_connect (td=0xc990e180, fd=89, sa=0xc98a1ba0) at
/usr/src/sys/kern/uipc_syscalls.c:536
#14 0xc070460f in connect (td=0xc990e180, uap=0xef781d04) at
/usr/src/sys/kern/uipc_syscalls.c:505
#15 0xc08f4193 in syscall (frame=
{tf_fs = 135725115, tf_es = 59, tf_ds = -1088487365, tf_edi =
135745024, tf_esi = -1089511444, tf_ebp = -1089514536, tf_isp = -277340828,
tf_ebx = 671753396, tf_edx = 0, tf_ecx = 135524256, tf_eax = 98, tf_trapno =
0, tf_err = 2, tf_eip = 674451435, tf_cs = 51, tf_eflags = 642, tf_esp =
-1089514580, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:984
#16 0xc08dde0f in Xint0x80_syscall () at
/usr/src/sys/i386/i386/exception.s:200
#17 0x0033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) f 7
#7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr =
0}, lport_arg=720306704, wild_okay=1)
at /usr/src/sys/netinet/in_pcb.c:923
923 in /usr/src/sys/netinet/in_pcb.c
(kgdb) i loc
phd = (struct inpcbport *) 0x2aef0210
tmphd = (struct inpcbport *) 0x2aef0210
match = (struct inpcb *) 0x0
inp = (struct inpcb *) 0x2aef0210
tmpinp = (struct inpcb *) 0x2aef0210
matchwild = 6
wildcard = -1062683820
lport = 14063
(kgdb) p phd
$1 = (struct inpcbport *) 0x2aef0210
(kgdb) p phd->phd_port
Cannot access memory at address 0x2aef021c
(kgdb) f 12
#12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180)
at /usr/src/sys/kern/uipc_socket.c:558
558 /usr/src/sys/kern/uipc_socket.c: No such file or directory.
in /usr/src/sys/kern/uipc_socket.c
(kgdb) p so
$2 = (struct socket *) 0xc97b39bc
(kgdb) p nam
$3 = (struct sockaddr *) 0xc98a1ba0
(kgdb) p td
$4 = (struct thread *) 0xc990e180
(kgdb) l
553 in /usr/src/sys/kern/uipc_socket.c
(kgdb) f 11
#11 0xc077e141 in tcp_usr_connect (so=0xc9897000, nam=0xc98a1ba0,
td=0xc990e1
Re: crash at in_pcb.c
On Wed, Oct 29, 2008 at 2:37 PM, Kip Macy <[EMAIL PROTECTED]> wrote: > The code in 7.0 is actually locked quite differently. Could you please > try and reproduce on 7.0 and RELENG_7? > > ok. I'll keep you posted. Jerry ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: crash at in_pcb.c
The code in 7.0 is actually locked quite differently. Could you please
try and reproduce on 7.0 and RELENG_7?
Thanks,
Kip
On Wed, Oct 29, 2008 at 8:45 PM, Jerry Toung <[EMAIL PROTECTED]> wrote:
> Hello List,
> I can realiably reproduce this crash. We have a deamon that accept several
> connections
> per sec. We use iperf and Microsoft Web application stress 1.0 to push
> traffic to the FreeBSD box.
> Without further delay, the crash dump is below. I've been troubleshooting,
> but I am no longer sure
> if this is a race condition or a stack corruption. The socket pointer
> between frame 12 and 11 is different.
> This is on 6.2, but the code for 7.0 is identical, so I think it still
> applies.
>
> Any hint, patching or troubleshooting this is appreciated.
>
> Unread portion of the kernel message buffer:
>
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address = 0x2aef0210
> fault code = supervisor read, page not present
> instruction pointer = 0x20:0xc0769098
> stack pointer = 0x28:0xef781bc0
> frame pointer = 0x28:0xef781bd0
> code segment= base 0x0, limit 0xf, type 0x1b
>= DPL 0, pres 1, def32 1, gran 1
> processor eflags= interrupt enabled, resume, IOPL = 0
> current process = 1166 (ndaemon)
> trap number = 12
> panic: page fault
> cpuid = 0
> Uptime: 8h32m25s
> Dumping 3325 MB (3 chunks)
> #0 doadump () at pcpu.h:165
> 165 pcpu.h: No such file or directory.
>in pcpu.h
> (kgdb) l *0xc0769098
> 0xc0769098 is in in_pcblookup_local (/usr/src/sys/netinet/in_pcb.c:923).
> 918 /usr/src/sys/netinet/in_pcb.c: No such file or directory.
>in /usr/src/sys/netinet/in_pcb.c
> (kgdb) bt
> #0 doadump () at pcpu.h:165
> #1 0xc06c2812 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:412
> #2 0xc06c2bbd in panic (fmt=0xc0940872 "%s") at
> /usr/src/sys/kern/kern_shutdown.c:573
> #3 0xc08f3e4e in trap_fatal (frame=0xef781b80, eva=720306704) at
> /usr/src/sys/i386/i386/trap.c:838
> #4 0xc08f3b57 in trap_pfault (frame=0xef781b80, usermode=0, eva=720306704)
> at /usr/src/sys/i386/i386/trap.c:745
> #5 0xc08f3745 in trap (frame=
> {tf_fs = -277348344, tf_es = 40, tf_ds = -913309656, tf_edi = 6,
> tf_esi = 0, tf_ebp = -277341232, tf_isp = -277341268, tf_ebx = -1062683820,
> tf_edx = 720306704, tf_ecx = 14063, tf_eax = 720306704, tf_trapno = 12,
> tf_err = 0, tf_eip = -1065971560, tf_cs = 32, tf_eflags = 66050, tf_esp = 0,
> tf_ss = -1062683820}) at /usr/src/sys/i386/i386/trap.c:435
> #6 0xc08dddba in calltrap () at /usr/src/sys/i386/i386/exception.s:139
> #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr =
> 0}, lport_arg=720306704, wild_okay=1)
>at /usr/src/sys/netinet/in_pcb.c:923
> #8 0xc0768452 in in_pcbbind_setup (inp=0xc97150b4, nam=0x36ef,
> laddrp=0xc97150ec, lportp=0xc97150ce, cred=0xc8726780)
>at /usr/src/sys/netinet/in_pcb.c:464
> #9 0xc0767f56 in in_pcbbind (inp=0xc97150b4, nam=0x2aef0210,
> cred=0xc8726780) at /usr/src/sys/netinet/in_pcb.c:240
> #10 0xc077f272 in tcp_connect (tp=0xc9897000, nam=0xc98a1ba0, td=0xc990e180)
> at /usr/src/sys/netinet/tcp_usrreq.c:864
> #11 0xc077e141 in tcp_usr_connect (so=0xc9897000, nam=0xc98a1ba0,
> td=0xc990e180)
>at /usr/src/sys/netinet/tcp_usrreq.c:369
> #12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180)
> at /usr/src/sys/kern/uipc_socket.c:558
> #13 0xc07046a8 in kern_connect (td=0xc990e180, fd=89, sa=0xc98a1ba0) at
> /usr/src/sys/kern/uipc_syscalls.c:536
> #14 0xc070460f in connect (td=0xc990e180, uap=0xef781d04) at
> /usr/src/sys/kern/uipc_syscalls.c:505
> #15 0xc08f4193 in syscall (frame=
> {tf_fs = 135725115, tf_es = 59, tf_ds = -1088487365, tf_edi =
> 135745024, tf_esi = -1089511444, tf_ebp = -1089514536, tf_isp = -277340828,
> tf_ebx = 671753396, tf_edx = 0, tf_ecx = 135524256, tf_eax = 98, tf_trapno =
> 0, tf_err = 2, tf_eip = 674451435, tf_cs = 51, tf_eflags = 642, tf_esp =
> -1089514580, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:984
> #16 0xc08dde0f in Xint0x80_syscall () at
> /usr/src/sys/i386/i386/exception.s:200
> #17 0x0033 in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> (kgdb) f 7
> #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr =
> 0}, lport_arg=720306704, wild_okay=1)
>at /usr/src/sys/netinet/in_pcb.c:923
> 923 in /usr/src/sys/netinet/in_pcb.c
> (kgdb) i loc
> phd = (struct inpcbport *) 0x2aef0210
> tmphd = (struct inpcbport *) 0x2aef0210
> match = (struct inpcb *) 0x0
> inp = (struct inpcb *) 0x2aef0210
> tmpinp = (struct inpcb *) 0x2aef0210
> matchwild = 6
> wildcard = -1062683820
> lport = 14063
> (kgdb) p phd
> $1 = (struct inpcbport *) 0x2aef0210
> (kgdb) p phd->phd_port
> Cannot access memory at address 0x2aef021c
>
> (kgdb) f 12
> #12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180)
> at /usr/src/sys/kern/uipc_socket.c:
Re: neophyte: tcsetattr() gives 22 error in i386, not in amd64?
>> Hi, >> >> I'm getting a 22 errno from tcsetattr() on 7-STABLE i386 in code which >> was working under 7-STABLE amd64. Serial device is a ucom (silabs >> cp2103). Permissions on /dev/cuaU0 look fine. Cutecom/Minicom >> appears to open the port without error... >> >>> I don't see anything obviously wrong, but I'd bet a bug related to >> >>> 32/64-bit types. Can you post a complete piece of code that can be >> >>> compiled and run and demonstrates the problem? Also, try compiling with >> >>> -Wall -W and investigate any warnings that are produced. >> >>> >> >>> By the way, errno 22 is EINVAL, "Invalid argument". perror() is your >> >>> friend. >> >> Strange freebsd doesnt document error numbers. On POSIX, errno 22 is >> >> EINVAL as well (documented in errno(3)). Is this applicable to freebsd? >> > >> > /usr/include/errno.h isn't documentation of error numbers? >> > If you're wanting to track down how/why tcsetattr(3) results in EINVAL, > using truss or ktrace might come in handy. Otherwise, you literally > will have to throw some debugging code into the ucom(4) driver to > try and figure out what function is kicking out code 22. Wow! truss is quite handy. I've located the problem, and am posting it for posterity: Someone was memset()'ing the termios struct to zero's, then setting the baudrate (setcfspeed) and a couple other things. Apparently this was not a canonical set of required members of the struct, because adding a tcgetattr(f, termio) right after the memset apparently pre-populated the thing correctly and now it works fine... Thanks for the leg up, Jeremy. Best, Steve ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: crash at in_pcb.c
2008/10/29 Jerry Toung <[EMAIL PROTECTED]>:
> Hello List,
> I can realiably reproduce this crash. We have a deamon that accept several
> connections
> per sec. We use iperf and Microsoft Web application stress 1.0 to push
> traffic to the FreeBSD box.
> Without further delay, the crash dump is below. I've been troubleshooting,
> but I am no longer sure
> if this is a race condition or a stack corruption. The socket pointer
> between frame 12 and 11 is different.
> This is on 6.2, but the code for 7.0 is identical, so I think it still
> applies.
>
> Any hint, patching or troubleshooting this is appreciated.
>
> Unread portion of the kernel message buffer:
>
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address = 0x2aef0210
> fault code = supervisor read, page not present
> instruction pointer = 0x20:0xc0769098
> stack pointer = 0x28:0xef781bc0
> frame pointer = 0x28:0xef781bd0
> code segment= base 0x0, limit 0xf, type 0x1b
>= DPL 0, pres 1, def32 1, gran 1
> processor eflags= interrupt enabled, resume, IOPL = 0
> current process = 1166 (ndaemon)
> trap number = 12
> panic: page fault
> cpuid = 0
> Uptime: 8h32m25s
> Dumping 3325 MB (3 chunks)
> #0 doadump () at pcpu.h:165
> 165 pcpu.h: No such file or directory.
>in pcpu.h
> (kgdb) l *0xc0769098
> 0xc0769098 is in in_pcblookup_local (/usr/src/sys/netinet/in_pcb.c:923).
> 918 /usr/src/sys/netinet/in_pcb.c: No such file or directory.
>in /usr/src/sys/netinet/in_pcb.c
> (kgdb) bt
> #0 doadump () at pcpu.h:165
> #1 0xc06c2812 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:412
> #2 0xc06c2bbd in panic (fmt=0xc0940872 "%s") at
> /usr/src/sys/kern/kern_shutdown.c:573
> #3 0xc08f3e4e in trap_fatal (frame=0xef781b80, eva=720306704) at
> /usr/src/sys/i386/i386/trap.c:838
> #4 0xc08f3b57 in trap_pfault (frame=0xef781b80, usermode=0, eva=720306704)
> at /usr/src/sys/i386/i386/trap.c:745
> #5 0xc08f3745 in trap (frame=
> {tf_fs = -277348344, tf_es = 40, tf_ds = -913309656, tf_edi = 6,
> tf_esi = 0, tf_ebp = -277341232, tf_isp = -277341268, tf_ebx = -1062683820,
> tf_edx = 720306704, tf_ecx = 14063, tf_eax = 720306704, tf_trapno = 12,
> tf_err = 0, tf_eip = -1065971560, tf_cs = 32, tf_eflags = 66050, tf_esp = 0,
> tf_ss = -1062683820}) at /usr/src/sys/i386/i386/trap.c:435
> #6 0xc08dddba in calltrap () at /usr/src/sys/i386/i386/exception.s:139
> #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr =
> 0}, lport_arg=720306704, wild_okay=1)
>at /usr/src/sys/netinet/in_pcb.c:923
> #8 0xc0768452 in in_pcbbind_setup (inp=0xc97150b4, nam=0x36ef,
> laddrp=0xc97150ec, lportp=0xc97150ce, cred=0xc8726780)
>at /usr/src/sys/netinet/in_pcb.c:464
> #9 0xc0767f56 in in_pcbbind (inp=0xc97150b4, nam=0x2aef0210,
> cred=0xc8726780) at /usr/src/sys/netinet/in_pcb.c:240
> #10 0xc077f272 in tcp_connect (tp=0xc9897000, nam=0xc98a1ba0, td=0xc990e180)
> at /usr/src/sys/netinet/tcp_usrreq.c:864
> #11 0xc077e141 in tcp_usr_connect (so=0xc9897000, nam=0xc98a1ba0,
> td=0xc990e180)
>at /usr/src/sys/netinet/tcp_usrreq.c:369
> #12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180)
> at /usr/src/sys/kern/uipc_socket.c:558
> #13 0xc07046a8 in kern_connect (td=0xc990e180, fd=89, sa=0xc98a1ba0) at
> /usr/src/sys/kern/uipc_syscalls.c:536
> #14 0xc070460f in connect (td=0xc990e180, uap=0xef781d04) at
> /usr/src/sys/kern/uipc_syscalls.c:505
> #15 0xc08f4193 in syscall (frame=
> {tf_fs = 135725115, tf_es = 59, tf_ds = -1088487365, tf_edi =
> 135745024, tf_esi = -1089511444, tf_ebp = -1089514536, tf_isp = -277340828,
> tf_ebx = 671753396, tf_edx = 0, tf_ecx = 135524256, tf_eax = 98, tf_trapno =
> 0, tf_err = 2, tf_eip = 674451435, tf_cs = 51, tf_eflags = 642, tf_esp =
> -1089514580, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:984
> #16 0xc08dde0f in Xint0x80_syscall () at
> /usr/src/sys/i386/i386/exception.s:200
> #17 0x0033 in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> (kgdb) f 7
> #7 0xc0769098 in in_pcblookup_local (pcbinfo=0x2aef0210, laddr={s_addr =
> 0}, lport_arg=720306704, wild_okay=1)
>at /usr/src/sys/netinet/in_pcb.c:923
> 923 in /usr/src/sys/netinet/in_pcb.c
> (kgdb) i loc
> phd = (struct inpcbport *) 0x2aef0210
> tmphd = (struct inpcbport *) 0x2aef0210
> match = (struct inpcb *) 0x0
> inp = (struct inpcb *) 0x2aef0210
> tmpinp = (struct inpcb *) 0x2aef0210
> matchwild = 6
> wildcard = -1062683820
> lport = 14063
> (kgdb) p phd
> $1 = (struct inpcbport *) 0x2aef0210
> (kgdb) p phd->phd_port
> Cannot access memory at address 0x2aef021c
>
> (kgdb) f 12
> #12 0xc06fec4e in soconnect (so=0xc97b39bc, nam=0xc98a1ba0, td=0xc990e180)
> at /usr/src/sys/kern/uipc_socket.c:558
> 558 /usr/src/sys/kern/uipc_socket.c: No such file or directory.
>in /usr/src/sys/kern/uipc_socket.c
> (kgdb) p so
> $2 = (struct socket
