Hello, Starting about a week ago, our primary webserver (then running FreeBSD 7.0) began crashing several times a day, typically during our higher-load times of day. We have since upgraded to 7.1p7, but continued to see the frequent crashes.
We are running an apache22 webserver with a lot of perl, logging via syslog-ng, and using IPSec in transport mode between the webserver and both the fileserver and logserver. Everything is IPv4. From uname: | FreeBSD mug.rescomp.berkeley.edu 7.1-RELEASE-p7 FreeBSD 7.1-RELEASE-p7 | #0: Wed Sep 2 17:56:59 PDT 2009 | r...@mug.rescomp.berkeley.edu:/usr/obj/usr/src/sys/GENERIC amd64 Some information that appears typical across many crashes: | Unread portion of the kernel message buffer: | | Fatal trap 27: stack fault while in kernel mode | cpuid = 0; apic id = 00 | instruction pointer = 0x8:0xffffffff80559fb4 | stack pointer = 0x10:0xffffffffae39faf0 | frame pointer = 0x10:0xf85ecc37f9239402 | code segment = base 0x0, limit 0xfffff, type 0x1b | = DPL 0, pres 1, long 1, def32 0, gran 1 | processor eflags = interrupt enabled, resume, IOPL = 0 | current process = 27 (em0 taskq) | trap number = 27 | panic: stack fault | cpuid = 0 | Uptime: 43m44s | Physical memory: 4082 MB | Dumping 361 MB: 346 330 314 298 282 266 250 234 218 202em0: watchdog timeout -- resetting | (kgdb) bt | #0 doadump () at pcpu.h:195 | #1 0x0000000000000004 in ?? () | #2 0xffffffff804bd9b9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 | #3 0xffffffff804bddc2 in panic (fmt=0x104 <Address 0x104 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:574 | #4 0xffffffff807b9f23 in trap_fatal (frame=0xffffff00012d66e0, eva=Variable "eva" is not available. | ) at /usr/src/sys/amd64/amd64/trap.c:764 | #5 0xffffffff807baa75 in trap (frame=0xffffffffae39fa40) at /usr/src/sys/amd64/amd64/trap.c:565 | #6 0xffffffff807a042e in calltrap () at /usr/src/sys/amd64/amd64/exception.S:209 | #7 0xffffffff80559fb4 in ether_input (ifp=0xffffff00012bf000, m=0xffffff0003576000) at /usr/src/sys/net/if_ethersubr.c:545 | #8 0xffffffff802bd645 in em_rxeof (adapter=0xffffffff80e4c000, count=99) at /usr/src/sys/dev/e1000/if_em.c:4539 | #9 0xffffffff802be55e in em_handle_rxtx (context=Variable "context" is not available. | ) at /usr/src/sys/dev/e1000/if_em.c:1702 | #10 0xffffffff804f2afd in taskqueue_run (queue=0xffffff00012c8c80) at /usr/src/sys/kern/subr_taskqueue.c:282 | #11 0xffffffff804f2da6 in taskqueue_thread_loop (arg=Variable "arg" is not available. | ) at /usr/src/sys/kern/subr_taskqueue.c:401 | #12 0xffffffff8049b2f3 in fork_exit (callout=0xffffffff804f2d40 <taskqueue_thread_loop>, arg=0xffffffff80e50588, frame=0xffffffffae39fc80) at /usr/src/sys/kern/kern_fork.c:804 | #13 0xffffffff807a07fe in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:455 | #14 0x0000000000000000 in ?? () | #15 0x0000000000000000 in ?? () | #16 0x0000000000000001 in ?? () [...] | (kgdb) source debug/gdb6 | (kgdb) frame 7 | #7 0xffffffff80559fb4 in ether_input (ifp=0xffffff00012bf000, m=0xffffff0003576000) at /usr/src/sys/net/if_ethersubr.c:545 | 545 eh = mtod(m, struct ether_header *); | (kgdb) info locals | eh = (struct ether_header *) 0xf85ecc37f9239402 | (kgdb) info args | ifp = (struct ifnet *) 0xffffff00012bf000 | m = (struct mbuf *) 0xffffff0003576000 | (kgdb) mbuf m | 0xffffff0003576000: 125 bytes ext 0xaf29dcb45d53e701 packet: 125 bytes received via em0 | 0xbb763383e10eda22Cannot access memory at address 0xbb763383e10eda3a | (kgdb) If anyone can provide some points on other things I can try to get useful data out of these core dumps, I'm open to it. We did decide to stop mounting NFS, upgrade to syslog-ng3 (which supports TLS), and revert the webserver back to a GENERIC kernel. Since booting the GENERIC kernel, the system has been up for nearly 2 days. Right now, we're logging via TLS to a temporary/testing logserver. That logserver is one of our default builds with IPSec. It is configured to forward logs over udp/syslog (via IPSec in transport mode) to our primary logserver. Within hours of beginning to pass the production webserver's logs through this temporary logserver (and thus having its syslog-ng forward to the primary logserver), the temporary logserver began exhibiting the same behavior that the webserver was previously showing. We're totally grasping at straws here, but it's looking like some kind of bug related to IPSec. Maybe related to long messages? High volume of messages? We would love to get this hammered out, so please let me know if there's any debugging we can perform or patches we can try. Thanks, -- Chris Cowart Network Technical Lead Network & Infrastructure Services, RSSP-IT UC Berkeley
pgptMo1rBPS4g.pgp
Description: PGP signature
