All,
We're experiencing an issue with relayd, in that at random times it will
just fail to relay HTTP and HTTPS sessions (happens in spurts, web
browsers see nothing in "view source"), and then will resume as if
nothing abnormal had occurred. The traffic does not reach the intended
web hosts when this happens (I can see the drop in traffic to them as
this is occurring, and there aren't any telling entries in their Apache
logs). Here is a sample from our logs, even starting relayd with two
verbose flags (-vv) doesn't give us any more of a hint than "session
failed":
Oct 29 11:05:04 fw01 relayd[20205]: relay httpproxy, session 125664 (3
active), 0, ***.***.130.2 -> 192.168.15.101:80, done
Oct 29 11:05:04 fw01 relayd[12154]: relay httpproxy, session 549446 (6
active), 0, ***.***.199.10 -> 192.168.15.102:80, last write (done)
Oct 29 11:05:04 fw01 relayd[12154]: relay httpproxy, session 549453 (6
active), 0, ***.***.66.206 -> 192.168.15.102:80, session failed
Oct 29 11:05:04 fw01 relayd[11172]: relay httpproxy, session 130130 (2
active), 0, ***.***.112.2 -> 192.168.15.101:80, session failed
Oct 29 11:05:04 fw01 relayd[8459]: relay httpproxy, session 2438 (1
active), 0, ***.***.66.206 -> 192.168.15.102:80, last write (done)
Oct 29 11:05:04 fw01 relayd[11172]: relay httpproxy, session 130132 (3
active), 0, ***.***.130.2 -> 192.168.15.101:80, last write (done)
Oct 29 11:05:04 fw01 relayd[11172]: relay httpproxy, session 130133 (3
active), 0, ***.***.10.2 -> 192.168.15.101:80, session failed
Oct 29 11:05:04 fw01 relayd[12154]: relay httpproxy, session 549454 (6
active), 0, ***.***.130.2 -> 192.168.15.101:80, session failed
Oct 29 11:05:04 fw01 relayd[20205]: relay httpproxy, session 125665 (3
active), 0, ***.***.130.2 -> 192.168.15.101:80, session failed
Oct 29 11:05:04 fw01 relayd[23592]: relay httpproxy, session 2738 (1
active), 0, ***.***.199.10 -> 192.168.15.102:80, session failed
Oct 29 11:05:04 fw01 relayd[20205]: relay httpproxy, session 125666 (3
active), 0, ***.***.130.2 -> 192.168.15.101:80, session failed
Oct 29 11:05:04 fw01 relayd[20205]: relay httpproxy, session 125663 (2
active), 0, ***.***.66.206 -> 192.168.15.102:80, last write (done)
Oct 29 11:05:04 fw01 relayd[12154]: relay httpproxy, session 549456 (6
active), 0, ***.***.87.22 -> 192.168.15.101:80, last write (done)
Oct 29 11:05:04 fw01 relayd[11172]: relay httpproxy, session 130134 (3
active), 0, ***.***.130.2 -> 192.168.15.101:80, last write (done)
Occasionally we'll see these failures as well, even though all hosts are
reporting as up:
Oct 29 13:41:43 fw01 relayd[25343]: relay httpproxy, session 96576 (7
active), 0, ***.***.192.80 -> :80, session failed
Oct 29 13:41:43 fw01 relayd[25343]: relay httpproxy, session 96577 (7
active), 0, ***.***.192.80 -> :80, session failed
Oct 29 13:41:43 fw01 relayd[25343]: relay httpproxy, session 96578 (7
active), 0, ***.***.209.214 -> :80, session failed
Oct 29 13:41:43 fw01 relayd[25343]: relay httpproxy, session 96579 (7
active), 0, ***.***.162.106 -> :80, session failed
Oct 29 13:41:43 fw01 relayd[25343]: relay httpproxy, session 96580 (7
active), 0, ***.***.209.214 -> :80, session failed
Oct 29 13:41:44 fw01 relayd[25343]: relay httpproxy, session 96581 (7
active), 0, ***.***.162.106 -> :80, session failed
As you can see, the sessions are failing to multiple web front-ends, and
it happens with both HTTPS and HTTP relays. As I said, this happens in
spurts, with lots of failures in a short time, followed by a period of
happy stability. These hosts show as up in "relayctl show summ", and
they don't go down or even flap while this is occurring. A reboot of the
machine did not help. Any help or ideas for further troubleshooting
would be great.
Here is my dmesg, this machine is running -stable:
OpenBSD 4.7 (GENERIC.MP) #0: Wed Aug 11 21:48:07 PDT 2010
r...@fw01.**************.net:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz ("GenuineIntel"
686-class) 2.94 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR
real mem = 2145406976 (2046MB)
avail mem = 2069958656 (1974MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 11/03/09, BIOS32 rev. 0 @ 0xfdb70,
SMBIOS rev. 2.5 @ 0x7fedf000 (39 entries)
bios0: vendor Phoenix Technologies LTD version "1.3a" date 11/03/2009
bios0: Supermicro X7SBi
acpi0 at bios0: rev 2
acpi0: tables DSDT FACP _MAR MCFG APIC BOOT SPCR ERST HEST BERT EINJ
SLIC SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT
acpi0: wakeup devices PXHA(S5) PEX_(S5) LAN_(S5) USB4(S5) USB5(S5)
USB7(S5) ESB2(S5) EXP1(S5) EXP5(S5) EXP6(S5) USB1(S5) USB2(S5) USB3(S5)
USB6(S5) ESB1(S5) PCIB(S5) KBC0(S1) MSE0(S1) COM1(S5) COM2(S5) PWRB(S3)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 266MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz ("GenuineIntel"
686-class) 2.94 GHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 3 pa 0xfecc0000, version 20, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 2 (PXHA)
acpiprt2 at acpi0: bus 3 (PEX_)
acpiprt3 at acpi0: bus 5 (EXP1)
acpiprt4 at acpi0: bus 13 (EXP5)
acpiprt5 at acpi0: bus 15 (EXP6)
acpiprt6 at acpi0: bus 17 (PCIB)
acpicpu0 at acpi0: C3, PSS
acpicpu1 at acpi0: C3, PSS
acpibtn0 at acpi0: PWRB
acpivideo0 at acpi0: IGD0
bios0: ROM list: 0xc0000/0x9000 0xc9000/0x1000 0xca000/0x1000
ipmi at mainbus0 not configured
cpu0: Enhanced SpeedStep 2934 MHz: speeds: 2933, 2667, 2400, 2133, 1867,
1600 MHz
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 3200/3210 Host" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 3200/3210 PCIE" rev 0x01: apic 2
int 16 (irq 5)
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci2 at ppb1 bus 2
"Intel IOxAPIC" rev 0x09 at pci1 dev 0 function 1 not configured
ppb2 at pci0 dev 6 function 0 "Intel 3210 PCIE" rev 0x01: apic 2 int 16
(irq 5)
pci3 at ppb2 bus 3
em0 at pci3 dev 0 function 0 "Intel PRO/1000 PT (82575EB)" rev 0x02:
apic 2 int 16 (irq 5), address 00:25:90:04:c7:00
em1 at pci3 dev 0 function 1 "Intel PRO/1000 PT (82575EB)" rev 0x02:
apic 2 int 17 (irq 10), address 00:25:90:04:c7:01
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 2 int
16 (irq 5)
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 2 int
17 (irq 10)
uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x02: apic 2 int
18 (irq 11)
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 2 int
18 (irq 11)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: apic 2 int
16 (irq 5)
pci4 at ppb3 bus 5
ppb4 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02: apic 2 int
16 (irq 5)
pci5 at ppb4 bus 13
em2 at pci5 dev 0 function 0 "Intel PRO/1000MT (82573E)" rev 0x03: apic
2 int 16 (irq 5), address 00:30:48:fa:ec:c8
ppb5 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02: apic 2 int
17 (irq 10)
pci6 at ppb5 bus 15
em3 at pci6 dev 0 function 0 "Intel PRO/1000MT (82573L)" rev 0x00: apic
2 int 17 (irq 10), address 00:30:48:fa:ec:c9
uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 2 int
23 (irq 10)
uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 2 int
22 (irq 11)
uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 2 int
18 (irq 11)
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 2 int
23 (irq 10)
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb6 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci7 at ppb6 bus 17
vga1 at pci7 dev 3 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
radeondrm0 at vga1: apic 2 int 22 (irq 11)
drm0 at radeondrm0
pciide0 at pci7 dev 4 function 0 "ITExpress IT8213F" rev 0x00: DMA
(unsupported), channel 0 wired to native-PCI, channel 1 wired to native-PCI
pciide0: using apic 2 int 23 (irq 10) for native-PCI interrupt
pciide0: channel 0 ignored (not responding; disabled or no drives?)
pciide0: channel 1 ignored (not responding; disabled or no drives?)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02: PM disabled
pciide1 at pci0 dev 31 function 2 "Intel 82801I SATA" rev 0x02: DMA,
channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide1: using apic 2 int 17 (irq 255) for native-PCI interrupt
wd0 at pciide1 channel 0 drive 0: <ST3160316CS>
wd0: 16-sector PIO, LBA48, 152627MB, 312581808 sectors
wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 6
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 2
int 17 (irq 10)
iic0 at ichiic0
lm1 at iic0 addr 0x2d: W83627HF
wbng0 at iic0 addr 0x2f: w83793g
spdmem0 at iic0 addr 0x50: 1GB DDR2 SDRAM ECC PC2-6400CL5
spdmem1 at iic0 addr 0x52: 1GB DDR2 SDRAM ECC PC2-6400CL5
pciide2 at pci0 dev 31 function 5 "Intel 82801I SATA" rev 0x02: DMA,
channel 0 wired to native-PCI, channel 1 wired to native-PCI
pciide2: using apic 2 int 18 (irq 11) for native-PCI interrupt
"Intel 82801I Thermal" rev 0x02 at pci0 dev 31 function 6 not configured
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
wbsio0 at isa0 port 0x2e/2: W83627HF rev 0x41
lm2 at wbsio0 port 0x290/8: W83627HF
lm1 detached
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
mtrr: Pentium Pro MTRR support
uhidev0 at uhub5 port 2 configuration 1 interface 0 "Alcor Micro product
0x3107" rev 1.10/1.00 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub5 port 2 configuration 1 interface 1 "Alcor Micro product
0x3107" rev 1.10/1.00 addr 2
uhidev1: iclass 3/0, 2 report ids
uhid0 at uhidev1 reportid 1: input=2, output=0, feature=0
uhid1 at uhidev1 reportid 2: input=1, output=0, feature=0
vscsi0 at root
scsibus0 at vscsi0: 256 targets
softraid0 at root
root on wd0a swap on wd0b dump on wd0b
--
Thanks,
Andrew Klettke
Optic Fusion NOC
253-830-2943
