Hello everyone. I'm seeing the mclpool limit reached error.
I'm intending on replacing a transparent firewall running OpenBSD3.6 with
one running 4.2, and in the testing phase I've noticed an interesting
problem.
The intention is to have traffic coming in on interface A (trunk0 - bge0 and
bge1 loadbalanced) bridging onto interface B (em0) and interface C (em2).
At the moment I have interface B connected and traffic is getting to the new
firewall through the old firewall. Of course the new one is seeing traffic
coming in on an interface it doesn't expect, but that's not the issue I want
to discuss.
With only interface B connected I end up eventually getting the mclpool
limit reached error.
Looks like as the queue gets filled up (it's got nowhere to go, of course)
when it drops packets it doesn't properly release the allocated memory.
Check out the vmstat 3 commands down.
I've already got the 004pf patch compiled in, and I just added the 005 patch
and rebuilt and there's the same behaviour.
I won't see this problem in production, but it could mean that if a link
goes down eventually the firewall will require a reboot before properly
functioning.
Any comments?
# uname -a
OpenBSD xx.xx.xx 4.2 GENERIC.MP#0 i386
# dmesg | tail -1
WARNING: mclpool limit reached; increase kern.maxclusters
# sysctl kern.maxclusters
kern.maxclusters=6144
# vmstat -m | grep -e Releases -e mclpl ; pfctl -vsq|grep -B2 50/ ; sleep 10
; vmstat -m | grep -e Releases -e mclpl ; pfctl -vsq|grep -B2 50/
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg
Idle
mclpl 2048 48971 111592 38411 5285 0 5285 5285 4
6144 4
queue dmz-low-priority on em2 bandwidth 5Mb cbq( borrow default )
[ pkts: 0 bytes: 0 dropped pkts: 7055 bytes: 845459
]
[ qlength: 50/ 50 borrows: 0 suspends: 0 ]
--
queue svr-low-priority on em1 bandwidth 5Mb cbq( borrow default )
[ pkts: 0 bytes: 0 dropped pkts: 7054 bytes: 845361
]
[ qlength: 50/ 50 borrows: 0 suspends: 0 ]
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg
Idle
mclpl 2048 49149 111592 38556 5304 0 5304 5304 4
6144 6
queue dmz-low-priority on em2 bandwidth 5Mb cbq( borrow default )
[ pkts: 0 bytes: 0 dropped pkts: 7083 bytes: 848704
]
[ qlength: 50/ 50 borrows: 0 suspends: 0 ]
--
queue svr-low-priority on em1 bandwidth 5Mb cbq( borrow default )
[ pkts: 0 bytes: 0 dropped pkts: 7082 bytes: 848606
]
[ qlength: 50/ 50 borrows: 0 suspends: 0 ]
# ifconfig|grep -e flags -e media -e trunkp
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33208
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (none)
em2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (none)
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (none)
bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (none)
bge1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
media: Ethernet autoselect (none)
enc0: flags=0<> mtu 1536
trunk0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
trunk: trunkproto loadbalance
trunkport bge1
trunkport bge0 master
media: Ethernet autoselect
bridge0: flags=41<UP,RUNNING> mtu 1500
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33208
# brconfig bridge0|grep -v 'flags=0'
bridge0: flags=41<UP,RUNNING>
priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto
rstp
designated: id 00:00:00:00:00:00 priority 0
trunk0 flags=3<LEARNING,DISCOVER>
port 9 ifpriority 0 ifcost 0
em0 flags=3<LEARNING,DISCOVER>
port 1 ifpriority 0 ifcost 0
em1 flags=3<LEARNING,DISCOVER>
port 2 ifpriority 0 ifcost 0
em2 flags=3<LEARNING,DISCOVER>
port 3 ifpriority 0 ifcost 0
em3 flags=100<SPAN>
Addresses (max cache: 100, timeout: 240):
# grep -e svr-low-priority -e dmz-low-priority pf-*
pf-dmz.conf:pass out on $dmz_if inet proto icmp from any to <dmz_ips>
icmp-type 8 code 0 queue dmz-low-priority
pf-dmz.conf:pass out on $dmz_if inet proto icmp from any to <dmz_ips>
icmp-type 11 code 0 queue dmz-low-priority
pf-dmz.conf:pass out on $dmz_if inet proto tcp from any to <dmz_www_ips>
port { 80 443 } queue dmz-low-priority
pf-ece.conf:pass out on $svr_if inet proto icmp from any to <svr_net>
icmp-type 8 code 0 queue svr-low-priority
pf-ece.conf:pass out on $svr_if inet proto icmp from any to <svr_net>
icmp-type 11 code 0 queue svr-low-priority
pf-ece.conf:pass out on $svr_if inet proto udp from any to <svr_ntp_ips>
port 123 queue svr-low-priority
pf-ece.conf:pass out on $svr_if inet proto { tcp udp } from any to
<svr_ntp_ips> port 53 queue svr-low-priority
pf-eng.conf:pass out on $svr_if inet proto icmp from any to <eng_ips>
icmp-type 8 code 0 queue svr-low-priority
pf-eng.conf:pass out on $svr_if inet proto icmp from any to <eng_ips>
icmp-type 11 code 0 queue svr-low-priority
pf-nix.conf:pass out on $svr_if inet proto tcp from any to <svr_nix_mirror>
port 80 queue svr-low-priority
pf-nix.conf:pass out on $svr_if inet proto tcp from any to <svr_nix_mirror>
port 873 queue svr-low-priority
# grep -e svr-low-priority -e dmz-low-priority pf.conf
pf.conf:altq on $dmz_if bandwidth 100Mb cbq queue { dmz-low-priority
dmz-infrastructure dmz-services }
pf.conf: queue dmz-low-priority priority 1 bandwidth 5% cbq(default borrow)
pf.conf:altq on $svr_if bandwidth 100Mb cbq queue { svr-low-priority svr-eng
svr-ece svr-nix svr-win }
pf.conf: queue svr-low-priority priority 1 bandwidth 5% cbq(default borrow)
# cat
/var/run/dmesg.boot
OpenBSD 4.2 (GENERIC.MP) #0: Thu Dec 13 09:50:39 UTC 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel(R) Pentium(R) D CPU 3.00GHz ("GenuineIntel" 686-class) 3.01 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,CNXT-ID,CX16,xTPR
real mem = 2146922496 (2047MB)
avail mem = 2068299776 (1972MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 09/04/06, BIOS32 rev. 0 @ 0xf0010,
SMBIOS rev. 2.3 @ 0xfbd70 (62 entries)
bios0: vendor American Megatrends Inc. version "1001" date 09/04/2006
bios0: ASUSTek Computer INC. P5M2
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
apm0: flags 30102 dobusy 0 doidle 1
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf5040/256 (14 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82801GH LPC" rev 0x00)
pcibios0: PCI bus #6 is the last bus
bios0: ROM list: 0xc0000/0x9000
mainbus0: Intel MP Specification (Version 1.4)
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 200 MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Pentium(R) D CPU 3.00GHz ("GenuineIntel" 686-class) 3.01 GHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,CNXT-ID,CX16,xTPR
mainbus0: bus 0 is type PCI
mainbus0: bus 1 is type PCI
mainbus0: bus 2 is type PCI
mainbus0: bus 3 is type PCI
mainbus0: bus 4 is type PCI
mainbus0: bus 5 is type PCI
mainbus0: bus 6 is type PCI
mainbus0: bus 7 is type ISA
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 3 pa 0xfec10000, version 20, 24 pins
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel E7230 MCH" rev 0xc0
ppb0 at pci0 dev 28 function 0 "Intel 82801GB PCIE" rev 0x01
pci_intr_map: bus 0 dev 28 func 0 pin 3; line 6
pci_intr_map: no MP mapping found
pci_intr_map: bus 0 dev 28 func 0 pin 4; line 5
pci_intr_map: no MP mapping found
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci2 at ppb1 bus 2
ppb2 at pci2 dev 1 function 0 "Pericom PI7C21P100 PCIX-PCIX" rev 0x01
pci3 at ppb2 bus 3
em0 at pci3 dev 4 function 0 "Intel PRO/1000MT QP (82546GB)" rev 0x03: apic
3 int 0 (irq 10), address 00:1b:21:08:24:50
em1 at pci3 dev 4 function 1 "Intel PRO/1000MT QP (82546GB)" rev 0x03: apic
3 int 1 (irq 11), address 00:1b:21:08:24:51
em2 at pci3 dev 6 function 0 "Intel PRO/1000MT QP (82546GB)" rev 0x03: apic
3 int 2 (irq 6), address 00:1b:21:08:24:52
em3 at pci3 dev 6 function 1 "Intel PRO/1000MT QP (82546GB)" rev 0x03: apic
3 int 3 (irq 5), address 00:1b:21:08:24:53
ppb3 at pci0 dev 28 function 4 "Intel 82801G PCIE" rev 0x01
pci_intr_map: bus 0 dev 28 func 4 pin 3; line 6
pci_intr_map: no MP mapping found
pci_intr_map: bus 0 dev 28 func 4 pin 4; line 5
pci_intr_map: no MP mapping found
pci4 at ppb3 bus 4
bge0 at pci4 dev 0 function 0 "Broadcom BCM5721" rev 0x21, BCM5750 C1
(0x4201): apic 2 int 16 (irq 10), address 00:0e:a6:f4:eb:82
brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
ppb4 at pci0 dev 28 function 5 "Intel 82801G PCIE" rev 0x01
pci_intr_map: bus 0 dev 28 func 5 pin 3; line 6
pci_intr_map: no MP mapping found
pci_intr_map: bus 0 dev 28 func 5 pin 4; line 5
pci_intr_map: no MP mapping found
pci5 at ppb4 bus 5
bge1 at pci5 dev 0 function 0 "Broadcom BCM5721" rev 0x21, BCM5750 C1
(0x4201): apic 2 int 17 (irq 11), address 00:0e:a6:f4:ea:ba
brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
uhci0 at pci0 dev 29 function 0 "Intel 82801GB USB" rev 0x01: apic 2 int 23
(irq 15)
uhci1 at pci0 dev 29 function 1 "Intel 82801GB USB" rev 0x01: apic 2 int 19
(irq 5)
ehci0 at pci0 dev 29 function 7 "Intel 82801GB USB" rev 0x01: apic 2 int 23
(irq 15)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0: Intel EHCI root hub, rev 2.00/1.00, addr 1
ppb5 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xe1
pci_intr_map: bus 0 dev 30 func 0 pin 1; line 11
pci_intr_map: no MP mapping found
pci_intr_map: bus 0 dev 30 func 0 pin 2; line 5
pci_intr_map: no MP mapping found
pci6 at ppb5 bus 6
vga1 at pci6 dev 2 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801GB LPC" rev 0x01: PM disabled
pciide0 at pci0 dev 31 function 1 "Intel 82801GB IDE" rev 0x01: DMA, channel
0 configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
pciide1 at pci0 dev 31 function 2 "Intel 82801GB SATA" rev 0x01: DMA,
channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide1: using apic 2 int 19 (irq 5) for native-PCI interrupt
wd0 at pciide1 channel 0 drive 0: <WDC WD800JD-00MSA1>
wd0: 16-sector PIO, LBA48, 76319MB, 156301488 sectors
wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5
ichiic0 at pci0 dev 31 function 3 "Intel 82801GB SMBus" rev 0x01: apic 2 int
19 (irq 11)
iic0 at ichiic0
iic0: addr 0x2f 00=80 05=10 06=bf 07=10 08=bf 09=64 0a=64 0b=5e 0c=73 0d=5c
0e=7b 0f=12 10=96 11=96 12=99 13=ff 14=5a 15=71 16=cd 17=7e 18=c9 19=cc
1a=c0 1b=3a 1c=38 1d=80 1e=80 1f=80 20=20 21=2b 22=02 23=01 24=e0 25=0f
27=0f 29=0f 2b=0f 2f=0f 31=0f 33=0f 3b=ff 3c=ff 3d=ff 3e=ff 3f=ff 40=09
44=40 4e=40 50=10 58=29 59=01 5c=0e 5d=01 5e=55 5f=03 60=ff 62=ff 64=ff
66=ff 67=ff 68=3f 6a=ff 6c=ff 6e=ff 70=ff 72=ff 74=ff 76=ff 78=64 79=5f
7a=55 7b=50 7c=64 7d=5f 7e=55 7f=50 80=64 81=5f 82=55 83=50 84=64 85=5f
86=55 87=50 88=64 89=5f 8a=55 8b=50 8c=64 8d=5f 8e=55 8f=50 90=ff 91=ff
92=ff 93=ff 94=ff 95=ff 96=ff 97=ff 98=ff 99=ff 9a=ff 9b=ff 9c=ff 9d=ff
9e=ff 9f=ff a0=ff a1=ff a2=ff a3=ff a4=ff a5=ff a6=ff a7=ff ae=ff af=ff
b0=08 b1=01 b2=3f b3=3f b4=3f b5=3f b6=3f b7=3f b8=3f b9=3f ba=3f bb=89
bc=89 bd=89 be=89 bf=89 c0=89 c1=89 c2=89 c3=02 c4=0c c5=50 c6=ff c9=ff
ca=ff cb=ff cc=ff cd=ff ce=ff cf=ff d1=46 d2=46 d3=46 d4=46 d6=f0 d7=ff
d8=80 d9=01 da=80 db=01 dc=80 dd=01 de=80 df=01 e0=bb e1=c0 e2=82 e3=fe
e4=80 e5=c6 e6=fb e8=ff e9=ff ea=ce eb=4a ec=ff ed=ff ee=ff ef=ff f6=60
f7=80 f8=1b fa=ff fd=10
iic0: addr 0x4b 00=39 02=5f 03=64 04=39 06=5f 07=64 08=39 0a=5f 0b=64 0c=39
0e=5f 0f=64 10=39 12=5f 13=64 14=39 16=5f 17=64 18=39 1a=5f 1b=64 1c=39
1e=5f 1f=64 20=39 22=5f 23=64 24=39 26=5f 27=64 28=39 2a=5f 2b=64 2c=39
2e=5f 2f=64 30=39 32=5f 33=64 34=39 36=5f 37=64 38=39 3a=5f 3b=64 3c=39
3e=5f 3f=64 40=39 42=5f 43=64 44=39 46=5f 47=64 48=39 4a=5f 4b=64 4c=39
4e=5f 4f=64 50=39 52=5f 53=64 54=39 56=5f 57=64 58=39 5a=5f 5b=64 5c=39
5e=5f 5f=64 60=39 62=5f 63=64 64=39 66=5f 67=64 68=39 6a=5f 6b=64 6c=39
6e=5f 6f=64 70=39 72=5f 73=64 74=39 76=5f 77=64 78=39 7a=5f 7b=64 7c=39
7e=5f 7f=64 80=39 82=5f 83=64 84=39 86=5f 87=64 88=39 8a=5f 8b=64 8c=39
8e=5f 8f=64 90=39 92=5f 93=64 94=39 96=5f 97=64 98=39 9a=5f 9b=64 9c=39
9e=5f 9f=64 a0=39 a2=5f a3=64 a4=39 a6=5f a7=64 a8=39 aa=5f ab=64 ac=39
ae=5f af=64 b0=39 b2=5f b3=64 b4=39 b6=5f b7=64 b8=39 ba=5f bb=64 bc=39
be=5f bf=64 c0=39 c2=5f c3=64 c4=39 c6=5f c7=64 c8=39 ca=5f cb=64 cc=39
ce=5f cf=64 d0=39 d2=5f d3=64 d4=39 d6=5f d7=64 d8=39 da=5f db=64 dc=39
de=5f df=64 e0=39 e2=5f e3=64 e4=39 e6=5f e7=64 e8=39 ea=5f eb=64 ec=39
ee=5f ef=64 f0=39 f2=5f f3=64 f4=39 f6=5f f7=64 f8=39 fa=5f fb=64 fc=39
fe=5f ff=64
usb1 at uhci0: USB revision 1.0
uhub1 at usb1: Intel UHCI root hub, rev 1.00/1.00, addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2: Intel UHCI root hub, rev 1.00/1.00, addr 1
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pctr: user-level cycle counter enabled
mtrr: Pentium Pro MTRR support
apm0: disconnected
dkcsum: wd0 matches BIOS drive 0x80
root on wd0a swap on wd0b dump on wd0b
