I've got a system with one 40GB PATA disk (wd0) where the system reside
and two 500GB SATA disks (wd1 and wd2 as sd0) in RAID-1 using softraid(4).
The system currently ran 4.5-stable but it would sometimes hang when
copying files from wd0 to sd0. I installed the latest snapshot and
compiled GENERIC.MP from a cvs update earlier today. It seemed fine
at first, but then it hung again.
I can ping the machine, but I cannot ssh to it nor log in via the
serial console. Sending a break gets me into ddb.
Any hints on what to try next?
Script started on Wed May 13 18:08:18 2009
$ cu -l cua02
Connected
~Stopped at Debugger+0x4: leave
ddb{0}> ps
PID PPID PGRP UID S FLAGS WAIT COMMAND
32679 857 32679 1000 3 0x2200002 buf_needva ksh
19918 10951 10951 1000 3 0x2004000 buf_needva scp
10951 6187 10951 1000 3 0x2004080 pause ksh
6187 8449 8449 1000 3 0x2000180 select sshd
8449 14764 8449 0 3 0x2004180 netio sshd
857 11805 857 1000 3 0x2004082 pause ksh
11805 31825 31825 1000 3 0x2000180 select sshd
31825 14764 31825 0 3 0x2004180 netio sshd
2480 1 2480 0 3 0x2004002 buf_needva getty
20642 1 20642 0 3 0x2004082 ttyin getty
31522 1 31522 0 3 0x2004082 ttyin getty
17346 1 17346 0 3 0x2004082 ttyin getty
25948 1 25948 0 3 0x2004082 ttyin getty
2961 1 2961 0 3 0x2004082 ttyin getty
7107 23799 23799 0 3 0x2000180 pause smbd
267 1 267 0 3 0x2000080 select cron
27138 1 27138 0 3 0x2000080 select nmbd
23799 1 23799 0 3 0x2000180 select smbd
14764 1 14764 0 3 0x2000080 select sshd
358 1 358 0 3 0x2040180 select sendmail
19482 1 19482 0 3 0x2000180 select inetd
32217 1 32217 0 3 0x2000000 buf_needva ntpd
31457 12657 31457 83 3 0x2000180 poll ntpd
12657 1 12657 83 3 0x2000180 poll ntpd
20548 28310 28310 0 3 0x2000080 nfsd nfsd
30939 28310 28310 0 3 0x2000080 nfsd nfsd
29710 28310 28310 0 3 0x2000080 nfsd nfsd
16857 28310 28310 0 3 0x2000080 nfsd nfsd
28310 1 28310 0 3 0x2000080 netcon nfsd
24951 1 24951 0 3 0x2000080 select mountd
32472 31013 31013 74 3 0x2000180 bpf pflogd
17051 1 17051 28 3 0x2000180 poll portmap
31013 1 31013 0 3 0x2000080 netio pflogd
10136 14092 14092 73 3 0x2000180 poll syslogd
14092 1 14092 0 3 0x2000088 netio syslogd
19 0 0 0 3 0x2100200 bored crypto
18 0 0 0 3 0x2100200 aiodoned aiodoned
17 0 0 0 3 0x2100200 buf_needva update
16 0 0 0 3 0x2100200 cleaner cleaner
15 0 0 0 3 0x100200 reaper reaper
14 0 0 0 3 0x2100200 pgdaemon pagedaemon
13 0 0 0 3 0x2100200 pftm pfpurge
12 0 0 0 3 0x2100200 usbevt usb4
11 0 0 0 3 0x2100200 usbevt usb3
10 0 0 0 3 0x2100200 usbevt usb2
9 0 0 0 3 0x2100200 usbevt usb1
8 0 0 0 3 0x2100200 usbtsk usbtask
7 0 0 0 3 0x2100200 usbevt usb0
6 0 0 0 3 0x2100200 acpi_idle acpi0
5 0 0 0 7 0x40100200 idle1
4 0 0 0 3 0x2100200 bored syswq
* 3 0 0 0 7 0x40100200 idle0
2 0 0 0 3 0x2100200 kmalloc kmthread
1 0 1 0 3 0x2004080 wait init
0 -1 0 0 3 0x2080200 scheduler swapper
ddb{0}> trace
Debugger(0,d8b402b8,d0844024,d0844018,d2c47800) at Debugger+0x4
comintr(d2c1e000) at comintr+0x2ea
Xintr_ioapic2() at Xintr_ioapic2+0x74
--- interrupt ---
cpu_idle_cycle(d08d3aa0) at cpu_idle_cycle+0xf
Bad frame pointer: 0xd097be78
ddb{0}> show registers
ds 0x4a0a0010
es 0xd0390010 bpf_catchpacket+0xb4
fs 0x58
gs 0xde220010 end+0xd94982c
edi 0xd2c1df40 end+0x234775c
esi 0xf9
ebp 0xde22bef0 end+0xd95570c
ebx 0xd2c1e4ac end+0x2347cc8
edx 0x3f8
ecx 0
eax 0x1
eip 0xd04954d4 Debugger+0x4
cs 0x50
eflags 0x202
esp 0xde22bef0 end+0xd95570c
ss 0xde220010 end+0xd94982c
Debugger+0x4: leave
ddb{0}> mach ddb 1
Stopped at Debugger+0x4: leave
ddb{1}> trace
Debugger(d2a70000,246,de22ff24,d0499cb0,d081f064) at Debugger+0x4
i386_ipi_handler(0,d0350058,d0820010,10,de220010) at i386_ipi_handler+0x4f
Xintripi() at Xintripi+0x47
--- interrupt ---
cpu_idle_cycle(d2a70000) at cpu_idle_cycle+0xf
Bad frame pointer: 0xd097be78
ddb{1}> show registers
ds 0xd04a0010 process_fninit_xmm+0x1c
es 0xde220010 end+0xd94982c
fs 0xd0350058 sensor_find+0xc
gs 0x10
edi 0xd2a70000 end+0x219981c
esi 0
ebp 0xde22fef4 end+0xd959710
ebx 0x6
edx 0xd0820f60 sched_idle_cpus
ecx 0x6
eax 0xffffffbf APTD+0xfbf
eip 0xd04954d4 Debugger+0x4
cs 0x50
eflags 0x246
esp 0xde22fef4 end+0xd959710
ss 0xde220010 end+0xd94982c
Debugger+0x4: leave
ddb{1}> show panic
the kernel did not panic
ddb{1}> boot reboot
~
[EOT]
$ # hangs at boot reboot
$ exit
Script done on Wed May 13 18:09:36 2009
Rebooting the system ...
Script started on Wed May 13 18:12:21 2009
$ cu -l cua02
Connected
>> OpenBSD/i386 BOOT 3.02
boot>
booting hd0a:/bsd: 6108448+1061284 [52+343264+325470]=0x779cac
entry point at 0x200120
[ using 669160 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2009 OpenBSD. All rights reserved. http://www.OpenBSD.org
OpenBSD 4.5-current (GENERIC.MP) #0: Wed May 13 17:16:04 CEST 2009
[email protected]:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Pentium(R) Dual-Core CPU E5200 @ 2.50GHz ("GenuineIntel" 686-class) 2.53
GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,EST,TM2,CX16,xTPR
real mem = 2146594816 (2047MB)
avail mem = 2067304448 (1971MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 09/18/08, BIOS32 rev. 0 @ 0xf0010, SMBIOS
rev. 2.5 @ 0xf06b0 (49 entries)
bios0: vendor American Megatrends Inc. version "0404" date 09/18/2008
bios0: ASUSTeK Computer INC. P5KPL-SE
acpi0 at bios0: rev 2
acpi0: tables DSDT FACP APIC MCFG OEMB HPET GSCI
acpi0: wakeup devices P0P2(S4) P0P1(S4) PS2K(S4) PS2M(S4) UAR1(S4) MC97(S4)
P0P4(S4) P0P5(S4) P0P6(S4) P0P7(S4) P0P8(S4) P0P9(S4) USB0(S4) USB1(S4)
USB2(S4) USB3(S4) EUSB(S4) SLPB(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 201MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Pentium(R) Dual-Core CPU E5200 @ 2.50GHz ("GenuineIntel" 686-class) 2.53
GHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,EST,TM2,CX16,xTPR
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 3 (P0P1)
acpiprt2 at acpi0: bus 2 (P0P4)
acpiprt3 at acpi0: bus -1 (P0P5)
acpiprt4 at acpi0: bus -1 (P0P6)
acpiprt5 at acpi0: bus 1 (P0P7)
acpicpu0 at acpi0
acpicpu1 at acpi0
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
bios0: ROM list: 0xc0000/0x8000
cpu0: unknown Enhanced SpeedStep CPU, msr 0x061a4c1c06004c1c
cpu0: using only highest and lowest power states
cpu0: Enhanced SpeedStep 15200 MHz (1148 mV): speeds: 15200, 1200 MHz
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x10
azalia0 at pci0 dev 27 function 0 "Intel 82801GB HD Audio" rev 0x01: apic 2 int
16 (irq 10)
azalia0: codecs: Realtek ALC662
audio0 at azalia0
ppb0 at pci0 dev 28 function 0 "Intel 82801GB PCIE" rev 0x01: apic 2 int 16
(irq 10)
pci1 at ppb0 bus 2
ppb1 at pci0 dev 28 function 3 "Intel 82801GB PCIE" rev 0x01: apic 2 int 19
(irq 11)
pci2 at ppb1 bus 1
re0 at pci2 dev 0 function 0 "Realtek 8168" rev 0x02: RTL8168C/8111C (0x3c00),
apic 2 int 19 (irq 11), address 00:23:54:bd:a5:44
rgephy0 at re0 phy 7: RTL8169S/8110S PHY, rev. 2
uhci0 at pci0 dev 29 function 0 "Intel 82801GB USB" rev 0x01: apic 2 int 23
(irq 5)
uhci1 at pci0 dev 29 function 1 "Intel 82801GB USB" rev 0x01: apic 2 int 19
(irq 11)
uhci2 at pci0 dev 29 function 2 "Intel 82801GB USB" rev 0x01: apic 2 int 18
(irq 3)
uhci3 at pci0 dev 29 function 3 "Intel 82801GB USB" rev 0x01: apic 2 int 16
(irq 10)
ehci0 at pci0 dev 29 function 7 "Intel 82801GB USB" rev 0x01: apic 2 int 23
(irq 5)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb2 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xe1
pci3 at ppb2 bus 3
vga1 at pci3 dev 0 function 0 "ATI Mach64" rev 0x3a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801GB LPC" rev 0x01: PM disabled
pciide0 at pci0 dev 31 function 1 "Intel 82801GB IDE" rev 0x01: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
wd0 at pciide0 channel 0 drive 0: <WDC WD400BB-00JHA0>
wd0: 16-sector PIO, LBA, 38166MB, 78165360 sectors
atapiscsi0 at pciide0 channel 0 drive 1
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <PLEXTOR, CD-R PX-W4824A, 1.03> ATAPI 5/cdrom
removable
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
cd0(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
pciide1 at pci0 dev 31 function 2 "Intel 82801GB SATA" rev 0x01: DMA, channel 0
configured to native-PCI, channel 1 configured to native-PCI
pciide1: using apic 2 int 19 (irq 11) for native-PCI interrupt
wd1 at pciide1 channel 0 drive 0: <SAMSUNG HD501LJ>
wd1: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
wd1(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5
wd2 at pciide1 channel 1 drive 0: <SAMSUNG HD501LJ>
wd2: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
wd2(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 5
ichiic0 at pci0 dev 31 function 3 "Intel 82801GB SMBus" rev 0x01: apic 2 int 19
(irq 11)
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 1GB DDR2 SDRAM non-parity PC2-6400CL5
spdmem1 at iic0 addr 0x52: 1GB DDR2 SDRAM non-parity PC2-6400CL5
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci3: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
wbsio0 at isa0 port 0x2e/2: W83627DHG rev 0x23
lm1 at wbsio0 port 0x290/8: W83627DHG
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
mtrr: Pentium Pro MTRR support
softraid0 at root
softraid0: trying to bring up sd0 degraded
scsibus1 at softraid0: 1 targets
sd0 at scsibus1 targ 0 lun 0: <OPENBSD, SR RAID 1, 003> SCSI2 0/direct fixed
sd0: 476937MB, 512 bytes/sec, 976767923 sec total
root on wd0a swap on wd0b dump on wd0b
WARNING: / was not properly unmounted
Automatic boot in progress: starting file system checks.
/dev/rwd0a: 107957 files, 700809 used, 2397066 free (1802 frags, 299408 blocks,
0.1% fragmentation)
/dev/rwd0a: MARKING FILE SYSTEM CLEAN
/dev/rsd0a: INCORRECT BLOCK COUNT I=1792896 (4 should be 0) (CORRECTED)
/dev/rsd0a: INCORRECT BLOCK COUNT I=1792897 (2048 should be 384) (CORRECTED)
/dev/rsd0a: INCORRECT BLOCK COUNT I=1792898 (2144 should be 384) (CORRECTED)
/dev/rsd0a: INCORRECT BLOCK COUNT I=1792899 (4 should be 0) (CORRECTED)
/dev/rsd0a: INCORRECT BLOCK COUNT I=1792900 (1760 should be 384) (CORRECTED)
[...]
/dev/rwd0d: 1208 files, 534006 used, 14561322 free (50 frags, 1820159 blocks,
0.0% fragmentation)
/dev/rwd0d: MARKING FILE SYSTEM CLEAN
/dev/rsd0a: LINK COUNT DIR I=2 OWNER=tpfaff MODE=40755
/dev/rsd0a: SIZE=512 MTIME=May 13 18:07 2009 COUNT 4 SHOULD BE 3 (ADJUSTED)
/dev/rsd0a: ZERO LENGTH DIR I=1792896 OWNER=tpfaff MODE=40755
/dev/rsd0a: SIZE=0 MTIME=May 13 18:07 2009 (CLEARED)
/dev/rsd0a: UNREF FILE I=1792897 OWNER=tpfaff MODE=100755
/dev/rsd0a: SIZE=1022810 MTIME=May 13 18:07 2009 (RECONNECTED)
/dev/rsd0a: NO lost+found DIRECTORY (CREATED)
/dev/rsd0a: UNREF FILE I=1792898 OWNER=tpfaff MODE=100755
/dev/rsd0a: SIZE=1068784 MTIME=May 13 18:07 2009 (RECONNECTED)
/dev/rsd0a: ZERO LENGTH DIR I=1792899 OWNER=tpfaff MODE=40755
/dev/rsd0a: SIZE=0 MTIME=May 13 18:07 2009 (CLEARED)
/dev/rsd0a: UNREF FILE I=1792900 OWNER=tpfaff MODE=100755
/dev/rsd0a: SIZE=870747 MTIME=May 13 18:07 2009 (RECONNECTED)
[... a gazillion of these ]
/dev/rsd0a: 118 files, 654728 used, 25148406 free (22 frags, 3143548 blocks,
0.0% fragmentation)
/dev/rsd0a: MARKING FILE SYSTEM CLEAN
/dev/rsd0d: 41741 files, 58492806 used, 156028169 free (3929 frags, 19503030
blocks, 0.0% fragmentation)
/dev/rsd0d: MARKING FILE SYSTEM CLEAN
setting tty flags
kbd: keyboard mapping set to no
pf enabled
ddb.panic: 1 -> 1
ddb.console: 0 -> 1
machdep.allowaperture: 0 -> 2
starting network
starting system logger
starting initial daemons: portmap mountd nfsd ntpd.
savecore: no core dump
checking quotas: done.
building ps databases: kvm dev.
clearing /tmp
starting pre-securelevel daemons:.
setting kernel security level: kern.securelevel: 0 -> 1
creating runtime link editor directory cache.
preserving editor files.
starting network daemons: sendmail inetd sshd.
starting local daemons: smbd nmbd.
standard daemons: cron.
Wed May 13 18:18:36 CEST 2009
OpenBSD/i386 (fs.tp76.info) (tty00)
login: tpfaff
Password:
Last login: Wed May 13 17:40:48 on ttyp0 from ws.tp76.info
OpenBSD 4.5-current (GENERIC.MP) #0: Wed May 13 17:16:04 CEST 2009
You have mail.
fs$ mount
/dev/wd0a on / type ffs (local, softdep)
/dev/wd0d on /home type ffs (local, nodev, nosuid, softdep)
/dev/sd0a on /backup type ffs (local, nodev, noexec, nosuid, softdep)
/dev/sd0d on /data type ffs (NFS exported, local, nodev, noexec, nosuid,
softdep)
fs$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/wd0a 5.9G 1.3G 4.3G 24% /
/dev/wd0d 28.8G 1.0G 26.3G 4% /home
/dev/sd0a 49.2G 1.2G 45.5G 3% /backup
/dev/sd0d 409G 112G 277G 29% /data
fs$ exit
OpenBSD/i386 (fs.tp76.info) (tty00)
login: ~
[EOT]
$ exit
Script done on Wed May 13 18:20:23 2009
