This morning, I discovered to my great astonishment that my machine had kernel-panicked in the middle of the night.
My main question is, how do you go about tracking something like this down? What utilities should I use to make sure my hardware is properly configured? Etc? If it happens again, should I write down all the output from the panic? On to the specifics: does anyone have any idea what might have happened? The last time this machine was rebooted was about a month ago, to add a USB 2 card and an internal drive. I also recompiled my kernel at that time to add USB 2 support. I may have made other changes; don't quite remember. Anyway, it's been chugging along quite happily since then, up until the aforementioned kernel panic. I rebooted and, a couple of minutes after the reboot, it panicked again, implicating exim. I rebooted again and it (meaning lilo? very early) complained of a bad CRC check. I rebooted it again and, a few seconds after finishing all startup scripts, it panicked, implicating a totally different app. I rebooted it again and wrangled it into single-user mode. I don't know whether it is significant that it didn't panick while I was in single-user mode for half an hour or so. I'd been running a kernel from the 2.4.21-4 source; 2.4.21-5 was available, so I recompiled the kernel using 2.4.21-5 and "make oldconfig". Since I've rebooted with that kernel, several hours ago, it's been well-behaved ... but geez, I'd really like to track down the source of the problem, since the machine was well-behaved for a month before having this problem, and for an extremely long time before that. Anyway, I have no way to know that whatever caused these kernel panics won't happen again. The only really odd thing I've noticed is the following in my boot log: Mon Oct 6 10:27:52 2003: Starting SpamAssassin Mail Filter Daemon: named-xfer[408]: can't make tmpfile (stub/fgm.com.fuRBwu): Inappropriate ioctl for device That's weird for a few reasons. 1) What does bind have to do with SA? 2) Inappropriate ioctl? Huh? Hrm, relevant data ... My mount output: /dev/hda4 on / type auto (rw,errors=remount-ro) proc on /proc type proc (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/hda1 on /boot type ext3 (rw) /dev/hda3 on /var type ext3 (rw) /dev/hdc1 on /var/mp3 type ext3 (rw) usbfs on /proc/bus/usb type usbfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) My boot log: Mon Oct 6 10:27:31 2003: bootlogd. Mon Oct 6 10:27:31 2003: Activating swap. Mon Oct 6 10:27:31 2003: Checking root file system... Mon Oct 6 10:27:31 2003: fsck 1.35-WIP (21-Aug-2003) Mon Oct 6 10:27:31 2003: /dev/hda4: clean, 185110/1909440 files, 1080140/3814902 blocks Mon Oct 6 10:27:31 2003: modprobe: modprobe: Can't locate module char-major-10-135 Mon Oct 6 10:27:31 2003: System time was Mon Oct 6 16:27:28 UTC 2003. Mon Oct 6 10:27:31 2003: Setting the System Clock using the Hardware Clock as reference... Mon Oct 6 10:27:31 2003: modprobe: modprobe: Can't locate module char-major-10-135 Mon Oct 6 10:27:31 2003: modprobe: modprobe: Can't locate module char-major-10-135 Mon Oct 6 10:27:31 2003: System Clock set. System local time is now Mon Oct 6 16:27:30 UTC 2003. Mon Oct 6 10:27:31 2003: Calculating module dependencies... done. Mon Oct 6 10:27:31 2003: Loading modules: sis900 modprobe: Can't locate module ethernet Mon Oct 6 10:27:31 2003: Mon Oct 6 10:27:31 2003: Checking all file systems... Mon Oct 6 10:27:31 2003: fsck 1.35-WIP (21-Aug-2003) Mon Oct 6 10:27:31 2003: /dev/hda1: clean, 38/16384 files, 9231/32728 blocks Mon Oct 6 10:27:31 2003: /dev/hda3: clean, 14464/524288 files, 109897/1048572 blocks Mon Oct 6 10:27:31 2003: /dev/hdc1: clean, 5371/10010624 files, 9661060/20010808 blocks Mon Oct 6 10:27:31 2003: Setting kernel variables.. Mon Oct 6 10:27:31 2003: Mounting local filesystems... Mon Oct 6 10:27:31 2003: mount: none already mounted or /dev/pts busy Mon Oct 6 10:27:31 2003: mount: according to mtab, devpts is already mounted on /dev/pts Mon Oct 6 10:27:31 2003: /dev/hda1 on /boot type ext3 (rw) Mon Oct 6 10:27:31 2003: /dev/hda3 on /var type ext3 (rw) Mon Oct 6 10:27:31 2003: /dev/hdc1 on /var/mp3 type ext3 (rw) Mon Oct 6 10:27:31 2003: Starting hotplug subsystem: usbsync:[001 001 001 001 001 ] . Mon Oct 6 10:27:49 2003: Cleaning: /etc/network/ifstate. Mon Oct 6 10:27:49 2003: Setting up IP spoofing protection: rp_filter. Mon Oct 6 10:27:49 2003: Configuring network interfaces... Starting mail retrieval agent: system-wide fetchmail not configured. Mon Oct 6 10:27:49 2003: Starting mail retrieval agent: system-wide fetchmail not configured. Mon Oct 6 10:27:49 2003: done. Mon Oct 6 10:27:49 2003: Portmap disabled per monique Mon Oct 6 10:27:49 2003: Loading the saved-state of the serial devices... Mon Oct 6 10:27:50 2003: /dev/ttyS0 at 0x03f8 (irq = 4) is a 16550A Mon Oct 6 10:27:50 2003: /dev/ttyS1 at 0x02f8 (irq = 3) is a 16550A Mon Oct 6 10:27:50 2003: ^[[9;30]^[[14;30] Mon Oct 6 10:27:50 2003: Setting the System Clock using the Hardware Clock as reference... Mon Oct 6 10:27:50 2003: modprobe: modprobe: Can't locate module char-major-10-135 Mon Oct 6 10:27:51 2003: System Clock set. Local time: Mon Oct 6 10:27:51 MDT 2003 Mon Oct 6 10:27:51 2003: Mon Oct 6 10:27:51 2003: Cleaning: /tmp /var/lock /var/run. Mon Oct 6 10:27:51 2003: Initializing random number generator... done. Mon Oct 6 10:27:51 2003: Recovering nvi editor sessions... done. Mon Oct 6 10:27:51 2003: Setting up X server socket directory /tmp/.X11-unix...done. Mon Oct 6 10:27:51 2003: INIT: Entering runlevel: 2 Mon Oct 6 10:27:52 2003: Starting hotplug subsystem:.... already started. process pending events. Mon Oct 6 10:27:52 2003: Starting kernel log daemon: klogd. Mon Oct 6 10:27:52 2003: exiting kerneld Mon Oct 6 10:27:52 2003: Starting domain name service: namednamed[385]: starting (/etc/bind/named.conf). named 8.4.1-P1-REL-NOESW Tue Aug 26 17:32:39 MDT 2003 Mon Oct 6 10:27:52 2003: [EMAIL PROTECTED]:/usr/local/src/Packages/bind/bind-8.4.1.0/src/bin/named Mon Oct 6 10:27:52 2003: named[385]: hint zone "" (IN) loaded (serial 0) Mon Oct 6 10:27:52 2003: named[385]: master zone "localhost" (IN) loaded (serial 1) Mon Oct 6 10:27:52 2003: named[385]: master zone "127.in-addr.arpa" (IN) loaded (serial 1) Mon Oct 6 10:27:52 2003: named[385]: master zone "0.in-addr.arpa" (IN) loaded (serial 1) Mon Oct 6 10:27:52 2003: named[385]: master zone "255.in-addr.arpa" (IN) loaded (serial 1) Mon Oct 6 10:27:52 2003: named[385]: listening on [127.0.0.1].53 (lo) Mon Oct 6 10:27:52 2003: named[385]: listening on [192.168.1.102].53 (eth0) Mon Oct 6 10:27:52 2003: named[385]: AF_INET6: address family not supported Mon Oct 6 10:27:52 2003: named[385]: Forwarding source address is [0.0.0.0].53 Mon Oct 6 10:27:52 2003: . Mon Oct 6 10:27:52 2003: named[405]: Ready to answer queries. Mon Oct 6 10:27:52 2003: Starting SpamAssassin Mail Filter Daemon: named-xfer[408]: can't make tmpfile (stub/fgm.com.fuRBwu): Inappropriate ioctl for device Mon Oct 6 10:27:53 2003: Mon Oct 6 10:27:57 2003: unix dgram connect: No such file or directory at /usr/sbin/spamd line 281 Mon Oct 6 10:27:57 2003: spamd. Mon Oct 6 10:27:58 2003: WARN: Bastille-firewall is not configured yet Mon Oct 6 10:27:58 2003: Please create /etc/Bastille/bastille-firewall.cfg to enable it. Mon Oct 6 10:27:58 2003: (hint: use InteractiveBastille) Mon Oct 6 10:27:58 2003: Enabling additional executable binary formats: binfmt-support. Mon Oct 6 10:27:58 2003: Mon Oct 6 10:27:58 2003: WARNING: portmapper inactive - RPC services unavailable! Mon Oct 6 10:27:58 2003: (Commenting out the rpc services in inetd.conf will Mon Oct 6 10:27:58 2003: disable this message) Mon Oct 6 10:27:58 2003: Mon Oct 6 10:27:58 2003: Starting internet superserver: inetd. Mon Oct 6 10:27:58 2003: Starting internet superserver: inetd. Mon Oct 6 10:27:58 2003: Starting system log daemon: syslogd. Mon Oct 6 10:27:59 2003: Starting printer spooler: lprng. Mon Oct 6 10:28:00 2003: Starting Samba daemons: nmbd smbd. Mon Oct 6 10:28:03 2003: Starting slimp3: slimp3. Mon Oct 6 10:28:05 2003: Starting OpenBSD Secure Shell server: sshd. Mon Oct 6 10:28:06 2003: Running ntpdate to synchronize clock. Mon Oct 6 10:28:07 2003: Starting NTP server: ntpd. Mon Oct 6 10:28:07 2003: Starting deferred execution scheduler: atd. Mon Oct 6 10:28:07 2003: Starting periodic command scheduler: cron. Mon Oct 6 10:28:07 2003: Started folding.home client Mon Oct 6 10:28:08 2003: Starting web server: apache. Mon Oct 6 10:28:11 2003: Starting mail retrieval agent: system-wide fetchmail not configured. Mon Oct 6 10:28:11 2003: Stopping Bootlog daemon: My dmesg: Linux version 2.4.21 ([EMAIL PROTECTED]) (gcc version 3.3.2 20030908 (Debian prerelease)) #2 Mon Oct 6 10:24:49 MDT 2003 BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 000000000009fc00 (usable) BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved) BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 000000001fff0000 (usable) BIOS-e820: 000000001fff0000 - 000000001fff8000 (ACPI data) BIOS-e820: 000000001fff8000 - 0000000020000000 (ACPI NVS) BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved) BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) BIOS-e820: 00000000ffee0000 - 00000000fff00000 (reserved) BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved) 511MB LOWMEM available. On node 0 totalpages: 131056 zone(0): 4096 pages. zone(1): 126960 pages. zone(2): 0 pages. Kernel command line: auto BOOT_IMAGE=linux ro root=304 Initializing CPU#0 Detected 1659.627 MHz processor. Console: colour VGA+ 80x43 Calibrating delay loop... 3316.12 BogoMIPS Memory: 516000k/524224k available (1358k kernel code, 7836k reserved, 505k data, 104k init, 0k highmem) Dentry cache hash table entries: 65536 (order: 7, 524288 bytes) Inode cache hash table entries: 32768 (order: 6, 262144 bytes) Mount cache hash table entries: 512 (order: 0, 4096 bytes) Buffer-cache hash table entries: 32768 (order: 5, 131072 bytes) Page-cache hash table entries: 131072 (order: 7, 524288 bytes) CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) CPU: L2 Cache: 256K (64 bytes/line) Intel machine check architecture supported. Intel machine check reporting enabled on CPU#0. CPU: After generic, caps: 0383f9ff c1c3f9ff 00000000 00000000 CPU: Common caps: 0383f9ff c1c3f9ff 00000000 00000000 CPU: AMD Athlon(tm) XP 2000+ stepping 02 Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Checking 'hlt' instruction... OK. POSIX conformance testing by UNIFIX PCI: PCI BIOS revision 2.10 entry at 0xfdb01, last bus=1 PCI: Using configuration type 1 PCI: Probing PCI hardware PCI: Using IRQ router SIS [1039/0008] at 00:02.0 isapnp: Scanning for PnP cards... isapnp: No Plug & Play device found Linux NET4.0 for Linux 2.4 Based upon Swansea University Computer Society NET3.039 Initializing RT netlink socket Starting kswapd Journalled Block Device driver loaded pty: 256 Unix98 ptys configured Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ SERIAL_PCI ISAPNP enabled ttyS00 at 0x03f8 (irq = 4) is a 16550A ttyS01 at 0x02f8 (irq = 3) is a 16550A Floppy drive(s): fd0 is 1.44M FDC 0 is a post-1991 82077 sis900.c: v1.08.06 9/24/2002 PCI: Assigned IRQ 11 for device 00:03.0 eth0: Realtek RTL8201 PHY transceiver found at address 1. eth0: Using transceiver found at address 1 as default eth0: SiS 900 PCI Fast Ethernet at 0xd400, IRQ 11, 00:0a:e6:52:7e:e7. Linux agpgart interface v0.99 (c) Jeff Hartmann agpgart: Maximum main memory to use for agp memory: 439M agpgart: Detected SiS 735 chipset agpgart: AGP aperture is 64M @ 0xd0000000 Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4 ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx hda: Maxtor 5T020H2, ATA DISK drive hdb: CREATIVE CD2423E, ATAPI CD/DVD-ROM drive hdc: Maxtor 6Y080P0, ATA DISK drive ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 ide1 at 0x170-0x177,0x376 on irq 15 hda: attached ide-disk driver. hda: host protected area => 1 hda: 40021632 sectors (20491 MB) w/2048KiB Cache, CHS=2491/255/63 hdc: attached ide-disk driver. hdc: host protected area => 1 hdc: 160086528 sectors (81964 MB) w/7936KiB Cache, CHS=158816/16/63 hdb: attached ide-cdrom driver. hdb: ATAPI 20X CD-ROM drive, 382kB Cache Uniform CD-ROM driver Revision: 3.12 Partition check: hda: hda1 hda2 hda3 hda4 hdc: hdc1 SCSI subsystem driver Revision: 1.00 kmod: failed to exec /sbin/modprobe -s -k scsi_hostadapter, errno = 2 kmod: failed to exec /sbin/modprobe -s -k scsi_hostadapter, errno = 2 usb.c: registered new driver usbdevfs usb.c: registered new driver hub PCI: Found IRQ 5 for device 00:0d.2 ehci-hcd 00:0d.2: NEC Corporation USB 2.0 ehci-hcd 00:0d.2: irq 5, pci mem e0816f00 usb.c: new USB bus registered, assigned bus number 1 PCI: 00:0d.2 PCI cache line size set incorrectly (32 bytes) by BIOS/FW. PCI: 00:0d.2 PCI cache line size corrected to 64. ehci-hcd 00:0d.2: USB 2.0 enabled, EHCI 0.95, driver 2003-Jan-22 hub.c: USB hub found hub.c: 5 ports detected host/usb-uhci.c: $Revision: 1.275 $ time 10:24:55 Oct 6 2003 host/usb-uhci.c: High bandwidth mode enabled host/usb-uhci.c: v1.275:USB Universal Host Controller Interface driver Initializing USB Mass Storage driver... usb.c: registered new driver usb-storage USB Mass Storage support registered. NET4: Linux TCP/IP 1.0 for NET4.0 IP: routing cache hash table of 4096 buckets, 32Kbytes TCP: Hash tables configured (established 32768 bind 65536) NET4: Unix domain sockets 1.0/SMP for Linux NET4.0. kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. VFS: Mounted root (ext3 filesystem) readonly. Freeing unused kernel memory: 104k freed Adding Swap: 524152k swap-space (priority -1) EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,4), internal journal kjournald starting. Commit interval 5 seconds EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,1), internal journal EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,3), internal journal EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds EXT3 FS 2.4-0.9.19, 19 August 2002 on ide1(22,1), internal journal EXT3-fs: mounted filesystem with ordered data mode. SiS router pirq escape (99) SiS router pirq escape (99) usb-ohci.c: USB OHCI at membase 0xe085f000, IRQ 12 usb-ohci.c: usb-00:02.3, Silicon Integrated Systems [SiS] 7001 (#2) usb.c: new USB bus registered, assigned bus number 2 hub.c: USB hub found hub.c: 3 ports detected PCI: Found IRQ 5 for device 00:02.2 PCI: Sharing IRQ 5 with 00:0d.1 usb-ohci.c: USB OHCI at membase 0xe0861000, IRQ 5 usb-ohci.c: usb-00:02.2, Silicon Integrated Systems [SiS] 7001 usb.c: new USB bus registered, assigned bus number 3 hub.c: USB hub found hub.c: 3 ports detected PCI: Found IRQ 5 for device 00:0d.1 PCI: Sharing IRQ 5 with 00:02.2 usb-ohci.c: USB OHCI at membase 0xe0863000, IRQ 5 usb-ohci.c: usb-00:0d.1, NEC Corporation USB (#2) usb.c: new USB bus registered, assigned bus number 4 hub.c: USB hub found hub.c: 2 ports detected PCI: Found IRQ 11 for device 00:0d.0 PCI: Sharing IRQ 11 with 00:02.7 usb-ohci.c: USB OHCI at membase 0xe0865000, IRQ 11 usb-ohci.c: usb-00:0d.0, NEC Corporation USB usb.c: new USB bus registered, assigned bus number 5 hub.c: USB hub found hub.c: 3 ports detected eth0: Media Link On 100mbps full-duplex -- monique Please respond to the group OR to my email, but not both. (Group preferred.) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
