Hi,Ok, those rl1: watchdog timeouts didn't ring a bell with me because I'd seen them before; however a quick grep in the logs (which date back to May 25) show no other watchdog timeout matches.
To try and avoid being incomplete again, I'll just attach the full dmesg below.
Jeremy Chadwick wrote:
On Wed, Aug 06, 2008 at 11:37:16AM +0200, Sebastiaan van Erk wrote:Yes, good thing you pointed this out, I hadn't seen those yet: Aug 5 11:15:05 piglet kernel: rl1: watchdog timeout Aug 5 11:15:05 piglet kernel: ad6: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=218885455 Aug 5 11:15:05 piglet kernel: ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=218885455 Aug 5 11:15:10 piglet kernel: rl1: watchdog timeout Aug 5 11:15:31 piglet kernel: rl1: watchdog timeout Aug 5 11:15:31 piglet kernel: ad6: FAILURE - device detached Aug 5 11:15:31 piglet kernel: subdisk6: detached Aug 5 11:15:31 piglet kernel: ad6: detached Aug 5 11:15:31 piglet kernel: rl1: watchdog timeout Aug 5 11:15:31 piglet kernel: rl1: watchdog timeout Aug 5 11:15:31 piglet kernel: ad4: FAILURE - device detached Aug 5 11:15:31 piglet kernel: subdisk4: detached Aug 5 11:15:31 piglet kernel: ad4: detached Aug 5 11:15:31 piglet kernel: GEOM_MIRROR: Device gm1: provider ad6 disconnected. Aug 5 11:15:31 piglet kernel: GEOM_MIRROR: Device gm1: provider ad4 disconnected. Aug 5 11:15:31 piglet kernel: GEOM_MIRROR: Device gm1: provider mirror/gm1 destroyed. Aug 5 11:15:31 piglet kernel: GEOM_MIRROR: Device gm1 destroyed. Aug 5 11:15:31 piglet kernel: g_vfs_done():mirror/gm1s1e[WRITE(offset=111376236544, length=16384)] error = 6Kudos to Andrey for asking a simple yet incredibly benefitial question. You have a much greater problem here, and it doesn't look specific to your disks. It looks as if an interrupt is stalled or locked. I'm willing to bet your rl1 Realtek NIC and your ATA controller (associated with disks ad4 and ad6) use the same IRQ. vmstat -i output should help clear that up, or dmesg output. I'll tell you that there have been some watchdog timeout fixes committed to rl(4) in recent months, depending upon what specific model and revision of Realtek NIC you have. No offence intended, but Realtek is definitely the worst of the bunch. I'm willing to bet it's an on-board NIC too. :-)
Actually, I have 3 NICs in my PC (all of them in use). My machine is the server/router in my home network, so it has the onboard vr0 NIC connected to my ADSL modem, the rl0 nic connected to my internal wired lan, and the rl1 nic connected to my wireless router (my internal wired lan is firewalled from the wireless, since I don't really trust wireless security ;-)).
I'm CC'ing PYUN Yong-Hyeon here, as he presently maintains/works on the rl(4) driver, and might be able to help determine if the Realtek NIC is what's causing all of this, or if the ATA chipset (is this the VIA? We don't know yet) is causing it first. Finally, what motherboard brand and model is this, and what BIOS revision or version?
I attached the output of dmidecode (and dmesg), hopefully that contains all you need to know.
BTW: I did a reply all, but I'm not sure if that is the "right" policy here. If I'm bothering anybody with this and they prefer to only see the mail on the list, then please let me know!
Regards and thanks for all the help, Sebastiaan
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.3-PRERELEASE #20: Wed Jan 2 19:48:49 CET 2008
[EMAIL PROTECTED]:/usr/obj/usr/src/sys/PIGLET
MPTable: <OEM00000 PROD00000000>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Sempron(tm) Processor 2600+ (1599.83-MHz 686-class CPU)
Origin = "AuthenticAMD" Id = 0x20fc2 Stepping = 2
Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
Features2=0x1<SSE3>
AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!>
AMD Features2=0x1<LAHF>
real memory = 1056964608 (1008 MB)
avail memory = 1020919808 (973 MB)
ioapic0: Assuming intbase of 0
ioapic0 <Version 0.3> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
hptrr: HPT RocketRAID controller driver v1.1 (Jan 2 2008 19:48:29)
cpu0 on motherboard
pcib0: <MPTable Host-PCI bridge> pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
agp0: <VIA 8380 host to PCI bridge> mem 0xe8000000-0xefffffff at device 0.0 on
pci0
pcib1: <MPTable PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
rl0: <RealTek 8139 10/100BaseTX> port 0xd000-0xd0ff mem 0xf6084000-0xf60840ff
irq 16 at device 8.0 on pci0
miibus0: <MII bus> on rl0
rlphy0: <RealTek internal media interface> on miibus0
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rl0: Ethernet address: 00:50:fc:57:a2:4b
rl1: <RealTek 8139 10/100BaseTX> port 0xd100-0xd1ff mem 0xf6080000-0xf60800ff
irq 17 at device 9.0 on pci0
miibus1: <MII bus> on rl1
rlphy1: <RealTek internal media interface> on miibus1
rlphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rl1: Ethernet address: 00:50:fc:44:23:0e
atapci0: <SiI SiI 3512 SATA150 controller> port
0xd200-0xd207,0xd300-0xd303,0xd400-0xd407,0xd500-0xd503,0xd600-0xd60f mem
0xf6081000-0xf60811ff irq 18 at device 10.0 on pci0
ata2: <ATA channel 0> on atapci0
ata3: <ATA channel 1> on atapci0
atapci1: <VIA 6420 SATA150 controller> port
0xd700-0xd707,0xd800-0xd803,0xd900-0xd907,0xda00-0xda03,0xdb00-0xdb0f,0xdc00-0xdcff
irq 20 at device 15.0 on pci0
ata4: <ATA channel 0> on atapci1
ata5: <ATA channel 1> on atapci1
atapci2: <VIA 8237 UDMA133 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xdd00-0xdd0f at device 15.1 on pci0
ata0: <ATA channel 0> on atapci2
ata1: <ATA channel 1> on atapci2
uhci0: <VIA 83C572 USB controller> port 0xde00-0xde1f irq 21 at device 16.0 on
pci0
uhci0: [GIANT-LOCKED]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xdf00-0xdf1f irq 21 at device 16.1 on
pci0
uhci1: [GIANT-LOCKED]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <VIA 83C572 USB controller> port 0xe000-0xe01f irq 21 at device 16.2 on
pci0
uhci2: [GIANT-LOCKED]
usb2: <VIA 83C572 USB controller> on uhci2
usb2: USB revision 1.0
uhub2: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <VIA 83C572 USB controller> port 0xe100-0xe11f irq 21 at device 16.3 on
pci0
uhci3: [GIANT-LOCKED]
usb3: <VIA 83C572 USB controller> on uhci3
usb3: USB revision 1.0
uhub3: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <VIA VT6202 USB 2.0 controller> mem 0xf6082000-0xf60820ff irq 21 at
device 16.4 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <VIA VT6202 USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
pcm0: <VIA VT8237> port 0xe200-0xe2ff irq 22 at device 17.5 on pci0
pcm0: <VIA Technologies VIA1617A AC97 Codec>
pcm0: <VIA DXS Enabled: DXS 4 / SGD 0 / REC 1>
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xe300-0xe3ff mem
0xf6083000-0xf60830ff irq 23 at device 18.0 on pci0
vr0: Quirks: 0x0
miibus2: <MII bus> on vr0
ukphy0: <Generic IEEE 802.3u media interface> on miibus2
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: Ethernet address: 00:13:d3:16:75:97
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xcc000-0xd3fff,0xd4000-0xd87ff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
unknown: <PNP0303> can't assign resources (port)
unknown: <PNP0c01> can't assign resources (memory)
unknown: <PNP0c02> can't assign resources (memory)
unknown: <PNP0501> can't assign resources (port)
unknown: <PNP0700> can't assign resources (port)
unknown: <PNP0400> can't assign resources (port)
unknown: <PNP0501> can't assign resources (port)
Timecounter "TSC" frequency 1599832423 Hz quality 800
Timecounters tick every 1.000 msec
hptrr: no controller detected.
ad0: 286188MB <Maxtor 6L300R0 BAH41G10> at ata0-master UDMA133
ad1: 239372MB <Maxtor 6L250R0 BAH41G10> at ata0-slave UDMA133
acd0: DMA limited to UDMA33, device found non-ATA66 cable
acd0: DVDR <LITE-ON DVDRW SHW-1635S/YS0N> at ata1-master UDMA33
ad4: 953869MB <SAMSUNG HD103UJ 1AA01112> at ata2-master SATA150
ad6: 953869MB <SAMSUNG HD103UJ 1AA01112> at ata3-master SATA150
ad8: 239372MB <Maxtor 6L250S0 BANC1G10> at ata4-master SATA150
ad10: 239372MB <Maxtor 6L250S0 BANC1G10> at ata5-master SATA150
GEOM_MIRROR: Device gm1 created (id=560014299).
GEOM_MIRROR: Device gm1: provider ad4 detected.
GEOM_MIRROR: Device gm1: provider ad6 detected.
GEOM_MIRROR: Device gm1: provider ad6 activated.
GEOM_MIRROR: Device gm1: provider mirror/gm1 launched.
GEOM_MIRROR: Device gm1: rebuilding provider ad4.
GEOM_MIRROR: Device gm0 created (id=1464068017).
GEOM_MIRROR: Device gm0: provider ad8 detected.
GEOM_MIRROR: Device gm0: provider ad10 detected.
GEOM_MIRROR: Device gm0: provider ad10 activated.
GEOM_MIRROR: Device gm0: provider ad8 activated.
GEOM_MIRROR: Device gm0: provider mirror/gm0 launched.
Trying to mount root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
/var: mount pending error: blocks 4 files 1
WARNING: /shared was not properly dismounted
WARNING: /mirror was not properly dismounted
WARNING: R/W mount of /data denied. Filesystem is not clean - run fsck
rl0: link state changed to UP
rl1: link state changed to UP
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
WARNING: R/W mount of /data denied. Filesystem is not clean - run fsck
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
ukbd0: Logitech HID compliant keyboard, rev 1.10/1.80, addr 2, iclass 3/1
kbd2 at ukbd0
uhid0: Logitech HID compliant keyboard, rev 1.10/1.80, addr 2, iclass 3/1
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
arplookup 169.254.8.225 failed: host is not on local network
ad4: FAILURE - SMART status=51<READY,DSC,ERROR> error=4<ABORTED>
ad6: FAILURE - SMART status=51<READY,DSC,ERROR> error=4<ABORTED>
ad4: FAILURE - SMART status=51<READY,DSC,ERROR> error=4<ABORTED>
ad4: FAILURE - SMART status=51<READY,DSC,ERROR> error=4<ABORTED>
ad6: FAILURE - SMART status=51<READY,DSC,ERROR> error=4<ABORTED>
GEOM_MIRROR: Device gm1: rebuilding provider ad4 finished.
GEOM_MIRROR: Device gm1: provider ad4 activated.
vr0: link state changed to DOWN
vr0: link state changed to UP
vr0: link state changed to DOWN
vr0: link state changed to UP
smime.p7s
Description: S/MIME Cryptographic Signature
