Please help if you can...
BACKGROUND
This crash is occurring on a dual-AMD 1.6Ghz cpu white-box system with 1 Gb
ram, 250Gb storage running GENERIC kernel. The system has been in production
use as a web server for nearly five years.
About 3 - 4 months ago, the system was upgraded from an earlier FreeBSD
version to 6.1. At the same time, all supporting applications (Apache
webserver, PERL, PostgreSQL, PHP, countless other applications & libraries)
were upgraded to the current releases. The system was stable up until a
couple of weeks ago.
FIRST ERROR EVENT
The system crashed during normal usage. The following message was displayed
on the console which was not responsive to keyboard input:
Sleeping thread (tid 100122, pid 11099)
owns a non-sleepable lock
panic: sleeping thread
cpuid=1
The system was restarted, an fsck routine was completed (answering "yes" to
all the "Do you want to salvage" type questions) and the server ran fine.
For about a week. It then crashed again several times, at intervals varying
from a few minutes of uptime to a few days.
SECOND ERROR EVENT
After some crashes, a message similar to that above was displayed. However,
at other times a message similar to this was displayed:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid=0; apic id=01
fault virtual address
=0x100
fault code
=supervisor read, page not present
instruction pointer
=0x20:0xc066c731
stack pointer
=0x28:0xe432ebf0
framepointer
=0x28:0xe432ebfc
code segment
=base 0x0, limit0xfffff, type 0x1b
=DPL 0, pres 1, def32 1, gran1
processor eflags
=resume, IOPL=0
current process
=36 (syncer)
trap number
= 12
panic: page fault
cpuid=0
uptime: 3d10h11m44s
Dumping 1535 Mb (2 chunks) [NOTE: the system had 1.5Gb memory at that
time. Memory was removed, reseated, swapped, etc., now 1Gb]
chunk 0:1Mb (159 pages)
CORRECTIONS ATTEMPTED
Somewhere during this ordeal, a Google search revealed a number of other
people experiencing the "Sleeping thread" problem. One of these was
apparently experienced in a FreeBSD 6.x development version stress test. No
definitive solution was identified in anything we say, except a single
reference to the problem being a kernel bug fixed in FreeBSD 6.2.
Accordingly, we upgraded from 6.1 to 6.2 but have still experienced the
problem.
We reviewed the 'messages' file and found references to several things which
led us to check FreeBSD 6.2 ERRATA
(http://www.freebsd.org/releases/6.2R/errata.html). This suggested adding
'kern.ipc.nmbclusters="0"' to the /boot/loader.conf file which might avoid a
known issue. We tried this, but saw no relief.
We also found a reference in the manual that suggested the issue might be a
problem with the APIC in 6.x. This recommended adding
'hint.apic.0.disabled="1"' to loader.conf. Tried this; no help.
In order to try to get more information about the system dumps we added:
dumpdev="AUTO" and dumpdir="/usr/crash" [to get more storage space than
available in /var/] and have generated several vmcore.# files of ~1 Gb each
(all identical size).
We attempted to use DDB to analyze the dumps (struggling now, unfamiliar
with kernel debugging process) with no success. Research suggested we needed
to create a debug version of the kernel (i.e., KERNEL.DEBUG) with debugging
options enabled.
We duly copied GENERIC and edited it, noting that "options ddb" was
already enabled. We added 'makeoptions DEBUG=-g # Build
kernel with gdb(1) debug symbols' as suggested and tried to "make
buildkernel" which errored out stating that KDB must be enabled to use DDB.
We edited KERNEL.DEBUG to add 'options KDB #
Enable kernel debugger' and attempted to "make buildkernel" again. This
time, the process stopped again with the message:
THIRD ERROR EVENT
[snip]
inline-unit-growth=100 --param
arge-function-growth=1000 -mno-align-long-strings -mpreferred-stack-boundary=2
-mno-mmx -mno-3dnow -mno-sse -mno-sse2 -ffreestanding -Werror
/usr/src/sys/crypto/sha2/sha2.c
/usr/src/sys/crypto/sha2/sha2.c: In function `SHA512_Transform':
/usr/src/sys/crypto/sha2/sha2.c:753: warning: 'T2' might be used
uninitialized in this function
*** Error code 1
Stop in /usr/obj/usr/src/sys/KERNEL.DEBUG.
*** Error code 1
Stop in /usr/src.
*** Error code 1
Stop in /usr/src.
www:/usr/src#
With this, we are stumped.
HELP PLEASE!
Can anyone:
- lead us to a solution based on these error messages?
- help us understand why the GENERIC kernel with only the debugging options
added failed to make?
- help us understand what '/usr/src/crypto/sha2/sha2.c' has to do with
anything?
- help us understand what we need to do to extract useful information from
the vmcore.# files?
- offer any other suggestions?
Thanks in advance!
_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
