Hi all,
After replacing all of the SCSI cables in my server and going for a week
without a panic I thought I'd gotten to the bottom of this. Now, for
the second morning in a row I've found this panic:
Debugger(d93328c4,d2bac000,df7e6ca0,d2bac000,d93445cc) at Debugger+0x4
ddb{0}> show panic
bad dir
ddb{0}>
ddb{0}> trace
Debugger(d93328c4,d2bac000,df7e6ca0,d2bac000,d93445cc) at Debugger+0x4
panic(d072703b,d93445cc,df7e6d40,d045a888,d93445cc) at panic+0x55
ufs_dirbad(d93445cc,1f800,d072702d,d8b65128,d9e61384) at ufs_dirbad+0x45
ufs_lookup(df7e6d58,d96799b4,df7e6d70,d039b1ff,d07fb438) at ufs_lookup+0xbc1
VOP_LOOKUP(d93328c4,df7e6e58,df7e6e6c,20) at VOP_LOOKUP+0x2e
lookup(df7e6e48,d9de4000,400,df7e6e60) at lookup+0x1d0
namei(df7e6e48,d9e0fbb8,0,cfc9b87c) at namei+0x18c
sys_stat(d96799b4,df7e6f68,df7e6f58,d96799b4) at sys_stat+0x4a
syscall() at syscall+0x12b
--- syscall (number 291) ---
0x6e6417d:
ddb{0}>
(ps and dmesg below)
Other details that might be of interest:
On reboot after this panic the LSI MegaRAID card will come up with a
mismatch between the stored configuration on the card and the
configuration information stored on the disk. I then have to enter the
LSI interface and choose to use the information stored on the card.
When the system then tries to bring up the disks, one of the RAID 1
volumes will be marked as failed. I then select one of the failed disk,
and bring it back on-line. This changes this volume from failed to
degraded, but when I do the same procedure for the other disk in the
RAID 1 volume, the alarms stop immediately and the volume is marked
on-line and optimal.
The system can then be rebooted and after fscking the disks be back to
normal, at least until tomorrow morning at about 1:30 if this pattern
holds :-(
I'll file this with sendbug as well.
Jeff
OpenBSD 4.4-current (GENERIC.MP) #1154: Fri Nov 14 16:40:08 MST 2008
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel(R) Xeon(TM) CPU 2.66GHz ("GenuineIntel" 686-class) 2.67 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS
,HTT,TM,SBF,CNXT-ID,xTPR
real mem = 2146988032 (2047MB)
avail mem = 2067472384 (1971MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 02/09/05, BIOS32 rev. 0 @ 0xf0010,
SMBIOS rev. 2.3 @ 0xf82a0 (48 entries)
bios0: vendor American Megatrends Inc. version "080008" date 02/09/2005
acpi0 at bios0: rev 0
acpi0: tables DSDT FACP APIC OEMB
acpi0: wakeup devices PS2K(S1) PS2M(S1) SMBS(S1) AUDI(S1) MODM(S1)
USB0(S1) USB1(S1) USB2(S1) P0P1(S1)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 133MHz
cpu1 at mainbus0: apid 6 (application processor)
cpu1: Intel(R) Xeon(TM) CPU 2.66GHz ("GenuineIntel" 686-class) 2.67 GHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS
,HTT,TM,SBF,CNXT-ID,xTPR
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Xeon(TM) CPU 2.66GHz ("GenuineIntel" 686-class) 2.67 GHz
cpu2:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS
,HTT,TM,SBF,CNXT-ID,xTPR
cpu3 at mainbus0: apid 7 (application processor)
cpu3: Intel(R) Xeon(TM) CPU 2.66GHz ("GenuineIntel" 686-class) 2.67 GHz
cpu3:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS
,HTT,TM,SBF,CNXT-ID,xTPR
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins
ioapic2 at mainbus0: apid 10 pa 0xfec80400, version 20, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (P0P1)
acpiprt2 at acpi0: bus 3 (P2P3)
acpiprt3 at acpi0: bus 5 (P2P4)
acpicpu0 at acpi0
acpicpu1 at acpi0
acpicpu2 at acpi0
acpicpu3 at acpi0
acpibtn0 at acpi0: SPBT
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1800 0xc9800/0x2200
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel E7501 Host" rev 0x01
ppb0 at pci0 dev 2 function 0 "Intel E7500 PCI" rev 0x01
pci1 at ppb0 bus 2
"Intel 82870P2 IOxAPIC" rev 0x04 at pci1 dev 28 function 0 not configured
ppb1 at pci1 dev 29 function 0 "Intel 82870P2 PCIX-PCIX" rev 0x04
pci2 at ppb1 bus 5
"Intel 82870P2 IOxAPIC" rev 0x04 at pci1 dev 30 function 0 not configured
ppb2 at pci1 dev 31 function 0 "Intel 82870P2 PCIX-PCIX" rev 0x04
pci3 at ppb2 bus 3
ppb3 at pci3 dev 3 function 0 "IBM 133 PCIX-PCIX" rev 0x03
pci4 at ppb3 bus 4
ami0 at pci4 dev 0 function 0 "Symbios Logic MegaRAID 320" rev 0x02:
apic 9 int 0 (irq 10)
ami0: LSI 532, 32b, FW 414C, BIOS vH429, 128MB RAM
ami0: 2 channels, 0 FC loops, 3 logical drives
scsibus0 at ami0: 40 targets, initiator 40
sd0 at scsibus0 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
sd0: 69618MB, 512 bytes/sec, 142577664 sec total
sd1 at scsibus0 targ 1 lun 0: <AMI, Host drive #01, > SCSI2 0/direct fixed
sd1: 69618MB, 512 bytes/sec, 142577664 sec total
sd2 at scsibus0 targ 2 lun 0: <AMI, Host drive #02, > SCSI2 0/direct fixed
sd2: 69618MB, 512 bytes/sec, 142577664 sec total
scsibus1 at ami0: 16 targets, initiator 16
safte0 at scsibus1 targ 6 lun 0: <SUPER, GEM318, 0> SCSI2 3/processor fixed
scsibus2 at ami0: 16 targets, initiator 16
safte1 at scsibus2 targ 6 lun 0: <SUPER, GEM318, 0> SCSI2 3/processor fixed
ahc0 at pci3 dev 6 function 0 "Adaptec AHA-29160 U160" rev 0x02: apic 9
int 4 (irq 10)
scsibus3 at ahc0: 16 targets, initiator 7
st0 at scsibus3 targ 6 lun 0: <SEAGATE, DAT 9SP40-000, 910B> SCSI3
1/sequential removable
uhci0 at pci0 dev 29 function 0 "Intel 82801CA/CAM USB" rev 0x02: apic 8
int 16 (irq 10)
ppb4 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x42
pci5 at ppb4 bus 1
fxp0 at pci5 dev 1 function 0 "Intel 8255x" rev 0x10, i82551: apic 8 int
17 (irq 5), address 00:e0:81:26:a9:e4
inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
vga1 at pci5 dev 2 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
skc0 at pci5 dev 4 function 0 "D-Link Systems DGE-530T A1" rev 0x11,
Yukon (0x1): apic 8 int 19 (irq 9)
sk0 at skc0 port A: address 00:13:46:72:3b:1d
eephy0 at sk0 phy 0: 88E1011 Gigabit PHY, rev. 3
ichpcib0 at pci0 dev 31 function 0 "Intel 82801CA LPC" rev 0x02
pciide0 at pci0 dev 31 function 1 "Intel 82801CA IDE" rev 0x02: DMA,
channel 0 configured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus4 at atapiscsi0: 2 targets, initiator 7
cd0 at scsibus4 targ 0 lun 0: <SONY, DVD RW DW-U18A, UYS4> ATAPI 5/cdrom
removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
ichiic0 at pci0 dev 31 function 3 "Intel 82801CA/CAM SMBus" rev 0x02:
apic 8 int 17 (irq 0)
iic0 at ichiic0
lm1 at iic0 addr 0x29: W83782D
spdmem0 at iic0 addr 0x50: 512MB DDR SDRAM registered ECC PC2100CL2.5
spdmem1 at iic0 addr 0x51: 512MB DDR SDRAM registered ECC PC2300CL2.5
spdmem2 at iic0 addr 0x54: 512MB DDR SDRAM registered ECC PC2100CL2.5
spdmem3 at iic0 addr 0x55: 512MB DDR SDRAM registered ECC PC2100CL2.5
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
wbsio0 at isa0 port 0x2e/2: W83627HF rev 0x3a
lm2 at wbsio0 port 0x290/8: W83627HF
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
mtrr: Pentium Pro MTRR support
softraid0 at root
root on sd0a swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
ddb{0}> ps
PID PPID PGRP UID S FLAGS WAIT COMMAND
*13142 6189 27098 1006 7 0x2004000 dspam
6189 30031 27098 1006 3 0x2000080 wait procmail
30031 30037 27098 1006 3 0x2004080 piperd procmail
30037 30984 27098 1006 3 0x2004080 wait preline
30984 22017 27098 1006 3 0x2004080 pause sh
22017 21447 27098 1006 3 0x2004080 wait qmail-local
11049 18360 27098 10001 3 0x2004080 select qmail-smtpd
7924 26085 27098 0 3 0x2004080 nanosleep sleep
17489 28341 27098 518 3 0x2004180 kqread imap-login
13242 28341 27098 518 3 0x2004180 kqread imap-login
10389 28341 27098 518 3 0x2004180 kqread imap-login
32730 28341 27098 1006 3 0x2044180 kqread imap
15279 28341 27098 518 3 0x2004180 kqread imap-login
9816 28341 27098 1006 2 0x2044580 imap
28208 28341 27098 518 3 0x2004180 kqread imap-login
1796 25706 1796 1006 3 0x2004082 ttyin ksh
25706 26514 26514 1006 3 0x2000180 select sshd
26514 19516 26514 0 3 0x2004180 netio sshd
21909 1 21909 0 3 0x2004082 ttyin getty
4369 28341 27098 1012 3 0x2044180 kqread imap
29495 28341 27098 518 3 0x2004180 kqread imap-login
14216 20109 14216 1021 3 0x2004082 ttyin ksh
1893 6705 1893 503 3 0x2000088 netio postgres
27942 6705 27942 503 3 0x2000088 netio postgres
24660 6705 24660 503 3 0x2000088 netio postgres
29170 6705 29170 503 3 0x2000088 netio postgres
2873 6705 2873 503 3 0x2000088 netio postgres
5323 6705 5323 503 3 0x2000088 netio postgres
10032 6705 10032 503 3 0x2000088 netio postgres
26384 28341 27098 1013 3 0x2044180 kqread imap
5669 28341 27098 1013 3 0x2044180 kqread imap
5809 28341 27098 518 3 0x2004180 kqread imap-login
6178 28341 27098 1013 3 0x2044180 kqread imap
13996 28341 27098 518 3 0x2004180 kqread imap-login
30138 28341 27098 1013 3 0x2044180 kqread imap
13394 28341 27098 518 3 0x2004180 kqread imap-login
18459 28341 27098 518 3 0x2004180 kqread imap-login
6547 6705 6547 503 3 0x2000088 netio postgres
13295 6705 13295 503 3 0x2000088 netio postgres
19220 6705 19220 503 3 0x2000088 netio postgres
7113 15471 15471 67 3 0x2000180 select httpd
13158 28341 27098 1006 3 0x2044180 kqread imap
25306 28341 27098 1006 3 0x2044180 kqread imap
9099 28341 27098 518 3 0x2004180 kqread imap-login
27690 28341 27098 518 3 0x2004180 kqread imap-login
13266 28341 27098 1006 3 0x2044180 kqread imap
4679 15471 15471 67 3 0x2000180 semwait httpd
22142 28341 27098 518 3 0x2004180 kqread imap-login
10512 28341 27098 0 3 0x2004080 kqread dovecot-auth
10309 15471 15471 67 3 0x2000180 semwait httpd
20040 15471 15471 67 3 0x2000180 semwait httpd
30895 15471 15471 67 3 0x2000180 semwait httpd
20109 1 20109 1006 3 0x2004082 pause ksh
19233 1 19233 1006 3 0x2004082 ttyin ksh
10448 1 10448 0 3 0x2004082 ttyin getty
23545 1 23545 0 3 0x2004082 ttyin getty
27196 1 27196 0 3 0x2004082 ttyin getty
24973 1 24973 0 3 0x2000080 poll wsmoused
10873 15471 15471 67 3 0x2000180 semwait httpd
433 15471 15471 67 3 0x2000180 semwait httpd
16518 15471 15471 67 3 0x2000180 semwait httpd
27669 15471 15471 67 3 0x2000180 semwait httpd
8463 15471 15471 67 3 0x2000180 semwait httpd
24629 1 24629 0 3 0x2000080 select cron
17184 1 17184 0 3 0x2080080 nanosleep sensorsd
15471 1 15471 67 3 0x2000180 select httpd
19516 1 19516 0 3 0x2000080 select sshd
9538 6705 9538 503 3 0x2000088 poll postgres
31056 6705 31056 503 2 0x2000488 postgres
32760 6705 32760 503 3 0x2000088 select postgres
25867 1 25867 28 3 0x2000180 poll portmap
498 794 27098 10004 3 0x2004082 piperd qmail-clean
10205 794 27098 10005 3 0x2004082 select qmail-rspawn
21447 794 27098 0 3 0x2004082 select qmail-lspawn
1784 5483 27098 83 3 0x2000182 poll ntpd
5483 7660 27098 83 3 0x2000182 poll ntpd
7660 21790 27098 0 3 0x2004082 poll ntpd
24553 28282 27098 32767 3 0x2004182 netio socklog
18360 15367 27098 10001 3 0x2004182 netcon tcpserver
19835 3478 27098 73 3 0x2004082 piperd multilog
11129 20992 27098 10008 3 0x2004082 piperd multilog
12840 14069 27098 532 3 0x2004082 piperd multilog
2484 5727 27098 77 3 0x2004182 poll dhcpd
654 3098 27098 0 2 0x2004082 socklog
2477 5309 27098 73 3 0x2004082 piperd multilog
24086 16330 27098 10002 3 0x2004082 piperd multilog
28341 20380 27098 0 3 0x2004082 kqread dovecot
595 16496 27098 10002 3 0x2004082 piperd multilog
11746 6905 27098 0 3 0x2004082 netcon sslserver
3887 16031 27098 73 3 0x2004082 piperd multilog
31468 7227 27098 0 3 0x2004082 piperd multilog
5063 5021 27098 10002 3 0x2004082 piperd multilog
20654 1578 27098 73 3 0x2004082 piperd multilog
21780 10255 27098 10008 3 0x2004182 netio tinydns
6705 9959 27098 503 3 0x200408a select postgres
19548 31103 27098 10002 3 0x2004082 piperd multilog
26521 28616 27098 10008 3 0x2004082 piperd multilog
8467 7683 27098 532 3 0x2004082 poll clamd
21790 9222 27098 0 3 0x2004082 pause sh
794 13047 27098 10006 3 0x2004082 select qmail-send
8979 16967 27098 0 3 0x2004082 netcon sslserver
14409 24327 27098 73 3 0x2004182 netio socklog
15640 28724 27098 10007 3 0x2004182 poll dnscache
14012 30344 27098 73 3 0x2004082 piperd multilog
1578 8210 27098 0 3 0x2004082 poll supervise
28282 8210 27098 0 3 0x2004082 poll supervise
16496 8210 27098 0 3 0x2004082 poll supervise
6905 8210 27098 0 3 0x2004082 poll supervise
7227 8210 27098 0 3 0x2004082 poll supervise
5727 8210 27098 0 3 0x2004082 poll supervise
16330 8210 27098 0 3 0x2004082 poll supervise
16967 8210 27098 0 3 0x2004082 poll supervise
5309 8210 27098 0 3 0x2004082 poll supervise
9222 8210 27098 0 3 0x2004082 poll supervise
30344 8210 27098 0 3 0x2004082 poll supervise
9959 8210 27098 0 3 0x2004082 poll supervise
20992 8210 27098 0 3 0x2004082 poll supervise
10255 8210 27098 0 3 0x2004082 poll supervise
26085 8210 27098 0 3 0x2004082 poll supervise
20380 8210 27098 0 3 0x2004082 poll supervise
14069 8210 27098 0 3 0x2004082 poll supervise
7683 8210 27098 0 3 0x2004082 poll supervise
3478 8210 27098 0 3 0x2004082 poll supervise
3098 8210 27098 0 3 0x2004082 poll supervise
31103 8210 27098 0 3 0x2004082 poll supervise
15367 8210 27098 0 3 0x2004082 poll supervise
16031 8210 27098 0 3 0x2004082 poll supervise
24327 8210 27098 0 3 0x2004082 poll supervise
5021 8210 27098 0 3 0x2004082 poll supervise
13047 8210 27098 0 3 0x2004082 poll supervise
28616 8210 27098 0 3 0x2004082 poll supervise
28724 8210 27098 0 3 0x2004082 poll supervise
739 27098 27098 0 3 0x2004082 piperd readproctitle
8210 27098 27098 0 3 0x2004082 nanosleep svscan
27098 1 27098 0 3 0x2004082 pause sh
17 0 0 0 3 0x2100200 bored crypto
16 0 0 0 3 0x2100200 aiodoned aiodoned
15 0 0 0 3 0x2100200 syncer update
14 0 0 0 3 0x2100200 cleaner cleaner
13 0 0 0 3 0x100200 reaper reaper
12 0 0 0 3 0x2100200 pgdaemon pagedaemon
11 0 0 0 3 0x2100200 pftm pfpurge
10 0 0 0 3 0x2100200 usbtsk usbtask
9 0 0 0 3 0x2100200 usbevt usb0
8 0 0 0 3 0x2100200 acpi_idle acpi0
7 0 0 0 7 0x100200 idle3
6 0 0 0 7 0x100200 idle2
5 0 0 0 7 0x100200 idle1
4 0 0 0 3 0x2100200 bored syswq
3 0 0 0 3 0x100200 idle0
2 0 0 0 3 0x2100200 kmalloc kmthread
1 0 1 0 3 0x2004080 wait init
0 -1 0 0 3 0x2080200 scheduler swapper
