Hello,
I have been running into a repeatable error reported by squid 4.6 from
packages once the system has been under a steady load for ~12 hours.
Example squid cache.log entry:
2019/05/22 15:03:41 kid1| oldAccept FD 18, 0.0.0.0 [ job2]: (53) Software
caused connection abort
The file descriptor given is always squid's main listening socket:
18 Socket 0 0 0 0.0.0.0:3128 HTTP Socket
Approx 120 clients using this system as their proxy.
Once this error hits, then web clients fail and squid needs to be restarted
to restore operation.
At first I had an LACP based trunk with the system's two em interfaces,
which I thought might be causing the issue (my first time using trunk). So
I switched to using the single bge interface but the errors still occur.
Can anyone suggest what I should be looking at to find the cause?
Thank you.
# netstat -in
Name Mtu Network Address Ipkts Ifail Opkts Ofail
Colls
lo0 32768 <Link> 66 0 66 0
0
lo0 32768 ::1/128 ::1 66 0 66 0
0
lo0 32768 fe80::%lo0/ fe80::1%lo0 66 0 66 0
0
lo0 32768 127/8 127.0.0.1 66 0 66 0
0
bge0 1500 <Link> d4:ae:52:c1:07:f2 32563772 0 34912639 0
0
bge0 1500 172.16.1/24 172.16.1.16 32563772 0 34912639 0
0
em0* 1500 <Link> 00:1b:21:5e:06:41 0 0 0 0
0
em1* 1500 <Link> 00:1b:21:5e:06:85 0 0 0 0
0
enc0* 0 <Link> 0 0 0 0
0
pflog0 33136 <Link> 0 0 0 0
0
# netstat -m
820 mbufs in use:
408 mbufs allocated to data
396 mbufs allocated to packet headers
16 mbufs allocated to socket names and addresses
398/1328 mbuf 2048 byte clusters in use (current/peak)
0/60 mbuf 2112 byte clusters in use (current/peak)
0/200 mbuf 4096 byte clusters in use (current/peak)
0/144 mbuf 8192 byte clusters in use (current/peak)
0/42 mbuf 9216 byte clusters in use (current/peak)
0/60 mbuf 12288 byte clusters in use (current/peak)
0/48 mbuf 16384 byte clusters in use (current/peak)
0/8 mbuf 65536 byte clusters in use (current/peak)
6452/7796/524288 Kbytes allocated to network (current/peak/max)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
OpenBSD 6.5 (GENERIC.MP) #0: Wed Apr 24 23:38:54 CEST 2019
r...@syspatch-65-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/
GENERIC.MP
real mem = 12865998848 (12269MB)
avail mem = 12466450432 (11888MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xf0450 (77 entries)
bios0: vendor Dell Inc. version "A16" date 07/06/2012
bios0: Dell Inc. Precision WorkStation T3500
acpi0 at bios0: rev 2
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC BOOT ASF! MCFG HPET TCPA DMAR SLIC SSDT
acpi0: wakeup devices VBTN(S4) PCI0(S5) PCI1(S5) PCI2(S5) PCI3(S5) PCI4(S5)
PCI5(S5) PCI6(S5) KBD_(S3) USB0(S3) USB1(S3) USB2(S3) USB3(S3) USB4(S3)
USB5(S3)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU W3565 @ 3.20GHz, 3333.80 MHz, 06-1a-05
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 133MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU W3565 @ 3.20GHz, 3333.28 MHz, 06-1a-05
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU W3565 @ 3.20GHz, 3333.28 MHz, 06-1a-05
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU W3565 @ 3.20GHz, 3333.28 MHz, 06-1a-05
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 1 (application processor)
cpu4: Intel(R) Xeon(R) CPU W3565 @ 3.20GHz, 3333.28 MHz, 06-1a-05
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 1, core 0, package 0
cpu5 at mainbus0: apid 3 (application processor)
cpu5: Intel(R) Xeon(R) CPU W3565 @ 3.20GHz, 3333.28 MHz, 06-1a-05
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 1, core 1, package 0
cpu6 at mainbus0: apid 5 (application processor)
cpu6: Intel(R) Xeon(R) CPU W3565 @ 3.20GHz, 3333.28 MHz, 06-1a-05
cpu6:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 1, core 2, package 0
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Xeon(R) CPU W3565 @ 3.20GHz, 3333.28 MHz, 06-1a-05
cpu7:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,MELTDOWN
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 1, core 3, package 0
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins, remapped
ioapic1 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins, remapped
acpimcfg0 at acpi0
acpimcfg0: addr 0xf8000000, bus 0-63
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 1 (PCI1)
acpiprt1 at acpi0: bus 2 (PCI2)
acpiprt2 at acpi0: bus 3 (PCI3)
acpiprt3 at acpi0: bus 4 (PCI4)
acpiprt4 at acpi0: bus 5 (PCI5)
acpiprt5 at acpi0: bus 6 (PCI6)
acpiprt6 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: !C3(350@96 mwait.1@0x20), !C2(500@64 mwait.1@0x10),
C1(1000@1 mwait.1)
acpicpu1 at acpi0: !C3(350@96 mwait.1@0x20), !C2(500@64 mwait.1@0x10),
C1(1000@1 mwait.1)
acpicpu2 at acpi0: !C3(350@96 mwait.1@0x20), !C2(500@64 mwait.1@0x10),
C1(1000@1 mwait.1)
acpicpu3 at acpi0: !C3(350@96 mwait.1@0x20), !C2(500@64 mwait.1@0x10),
C1(1000@1 mwait.1)
acpicpu4 at acpi0: !C3(350@96 mwait.1@0x20), !C2(500@64 mwait.1@0x10),
C1(1000@1 mwait.1)
acpicpu5 at acpi0: !C3(350@96 mwait.1@0x20), !C2(500@64 mwait.1@0x10),
C1(1000@1 mwait.1)
acpicpu6 at acpi0: !C3(350@96 mwait.1@0x20), !C2(500@64 mwait.1@0x10),
C1(1000@1 mwait.1)
acpicpu7 at acpi0: !C3(350@96 mwait.1@0x20), !C2(500@64 mwait.1@0x10),
C1(1000@1 mwait.1)
acpibtn0 at acpi0: VBTN
"*pnp0c14" at acpi0 not configured
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel X58 Host" rev 0x22
ppb0 at pci0 dev 1 function 0 "Intel X58 PCIE" rev 0x22: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 3 function 0 "Intel X58 PCIE" rev 0x22: msi
pci2 at ppb1 bus 2
vga1 at pci2 dev 0 function 0 vendor "NVIDIA", unknown product 0x040e rev
0xa1
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb2 at pci0 dev 7 function 0 "Intel X58 PCIE" rev 0x22: msi
pci3 at ppb2 bus 3
"Intel X58 Misc" rev 0x22 at pci0 dev 20 function 0 not configured
"Intel X58 GPIO" rev 0x22 at pci0 dev 20 function 1 not configured
"Intel X58 RAS" rev 0x22 at pci0 dev 20 function 2 not configured
uhci0 at pci0 dev 26 function 0 "Intel 82801JI USB" rev 0x00: apic 8 int 16
uhci1 at pci0 dev 26 function 1 "Intel 82801JI USB" rev 0x00: apic 8 int 17
uhci2 at pci0 dev 26 function 2 "Intel 82801JI USB" rev 0x00: apic 8 int 22
ehci0 at pci0 dev 26 function 7 "Intel 82801JI USB" rev 0x00: apic 8 int 22
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
ppb3 at pci0 dev 28 function 0 "Intel 82801JI PCIE" rev 0x00: msi
pci4 at ppb3 bus 4
ppb4 at pci0 dev 28 function 5 "Intel 82801JI PCIE" rev 0x00
pci5 at ppb4 bus 5
bge0 at pci5 dev 0 function 0 "Broadcom BCM5761" rev 0x10, BCM5761 A1
(0x5761100): msi, address d4:ae:52:c1:07:f2
brgphy0 at bge0 phy 1: BCM5761 10/100/1000baseT PHY, rev. 0
uhci3 at pci0 dev 29 function 0 "Intel 82801JI USB" rev 0x00: apic 8 int 23
uhci4 at pci0 dev 29 function 1 "Intel 82801JI USB" rev 0x00: apic 8 int 17
uhci5 at pci0 dev 29 function 2 "Intel 82801JI USB" rev 0x00: apic 8 int 18
ehci1 at pci0 dev 29 function 7 "Intel 82801JI USB" rev 0x00: apic 8 int 23
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
ppb5 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x90
pci6 at ppb5 bus 6
em0 at pci6 dev 4 function 0 "Intel 82541GI" rev 0x05: apic 8 int 16,
address 00:1b:21:5e:06:41
em1 at pci6 dev 5 function 0 "Intel 82541GI" rev 0x05: apic 8 int 17,
address 00:1b:21:5e:06:85
pcib0 at pci0 dev 31 function 0 "Intel 82801JIR LPC" rev 0x00
ahci0 at pci0 dev 31 function 2 "Intel 82801JI AHCI" rev 0x00: msi, AHCI 1.2
ahci0: port 0: 3.0Gb/s
ahci0: port 1: 3.0Gb/s
ahci0: PHY offline on port 2
ahci0: PHY offline on port 3
ahci0: port 4: 1.5Gb/s
ahci0: PHY offline on port 5
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, Crucial_CT275MX3, M0C> SCSI3 0/direct
fixed naa.500a07511481d376
sd0: 262321MB, 512 bytes/sector, 537234768 sectors, thin
sd1 at scsibus1 targ 1 lun 0: <ATA, Crucial_CT275MX3, M0C> SCSI3 0/direct
fixed naa.500a07511481de2a
sd1: 262321MB, 512 bytes/sector, 537234768 sectors, thin
cd0 at scsibus1 targ 4 lun 0: <HL-DT-ST, DVD+-RW GH82N, A102> ATAPI 5/cdrom
removable
ichiic0 at pci0 dev 31 function 3 "Intel 82801JI SMBus" rev 0x00: apic 8
int 20
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 4GB DDR3 SDRAM PC3-10600
spdmem1 at iic0 addr 0x52: 4GB DDR3 SDRAM PC3-10600
spdmem2 at iic0 addr 0x54: 4GB DDR3 SDRAM PC3-10600
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006> SCSI2 0/direct fixed
sd2: 262318MB, 512 bytes/sector, 537229073 sectors
root on sd2a (c1763ffc72f5bf56.a) swap on sd2b dump on sd2b
wsdisplay0: screen 5 deleted
wsdisplay0: screen 5 added (80x50, vt100 emulation)
