mclpool limit reached - pf queue dropping
Hello everyone. I'm seeing the mclpool limit reached error.
I'm intending on replacing a transparent firewall running OpenBSD3.6 with
one running 4.2, and in the testing phase I've noticed an interesting
problem.
The intention is to have traffic coming in on interface A (trunk0 - bge0 and
bge1 loadbalanced) bridging onto interface B (em0) and interface C (em2).
At the moment I have interface B connected and traffic is getting to the new
firewall through the old firewall. Of course the new one is seeing traffic
coming in on an interface it doesn't expect, but that's not the issue I want
to discuss.
With only interface B connected I end up eventually getting the mclpool
limit reached error.
Looks like as the queue gets filled up (it's got nowhere to go, of course)
when it drops packets it doesn't properly release the allocated memory.
Check out the vmstat 3 commands down.
I've already got the 004pf patch compiled in, and I just added the 005 patch
and rebuilt and there's the same behaviour.
I won't see this problem in production, but it could mean that if a link
goes down eventually the firewall will require a reboot before properly
functioning.
Any comments?
# uname -a
OpenBSD xx.xx.xx 4.2 GENERIC.MP#0 i386
# dmesg | tail -1
WARNING: mclpool limit reached; increase kern.maxclusters
# sysctl kern.maxclusters
kern.maxclusters=6144
# vmstat -m | grep -e Releases -e mclpl ; pfctl -vsq|grep -B2 50/ ; sleep 10
; vmstat -m | grep -e Releases -e mclpl ; pfctl -vsq|grep -B2 50/
NameSize Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg
Idle
mclpl 204848971 111592 38411 5285 0 5285 5285 4
61444
queue dmz-low-priority on em2 bandwidth 5Mb cbq( borrow default )
[ pkts: 0 bytes: 0 dropped pkts: 7055 bytes: 845459
]
[ qlength: 50/ 50 borrows: 0 suspends: 0 ]
--
queue svr-low-priority on em1 bandwidth 5Mb cbq( borrow default )
[ pkts: 0 bytes: 0 dropped pkts: 7054 bytes: 845361
]
[ qlength: 50/ 50 borrows: 0 suspends: 0 ]
NameSize Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg
Idle
mclpl 204849149 111592 38556 5304 0 5304 5304 4
61446
queue dmz-low-priority on em2 bandwidth 5Mb cbq( borrow default )
[ pkts: 0 bytes: 0 dropped pkts: 7083 bytes: 848704
]
[ qlength: 50/ 50 borrows: 0 suspends: 0 ]
--
queue svr-low-priority on em1 bandwidth 5Mb cbq( borrow default )
[ pkts: 0 bytes: 0 dropped pkts: 7082 bytes: 848606
]
[ qlength: 50/ 50 borrows: 0 suspends: 0 ]
# ifconfig|grep -e flags -e media -e trunkp
lo0: flags=8049 mtu 33208
em0: flags=8943 mtu 1500
media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
em1: flags=8943 mtu 1500
media: Ethernet autoselect (none)
em2: flags=8943 mtu 1500
media: Ethernet autoselect (none)
em3: flags=8843 mtu 1500
media: Ethernet autoselect (none)
bge0: flags=8943 mtu 1500
media: Ethernet autoselect (none)
bge1: flags=8943 mtu 1500
media: Ethernet autoselect (none)
enc0: flags=0<> mtu 1536
trunk0: flags=8943 mtu 1500
trunk: trunkproto loadbalance
trunkport bge1
trunkport bge0 master
media: Ethernet autoselect
bridge0: flags=41 mtu 1500
pflog0: flags=141 mtu 33208
# brconfig bridge0|grep -v 'flags=0'
bridge0: flags=41
priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto
rstp
designated: id 00:00:00:00:00:00 priority 0
trunk0 flags=3
port 9 ifpriority 0 ifcost 0
em0 flags=3
port 1 ifpriority 0 ifcost 0
em1 flags=3
port 2 ifpriority 0 ifcost 0
em2 flags=3
port 3 ifpriority 0 ifcost 0
em3 flags=100
Addresses (max cache: 100, timeout: 240):
# grep -e svr-low-priority -e dmz-low-priority pf-*
pf-dmz.conf:pass out on $dmz_if inet proto icmp from any to
icmp-type 8 code 0 queue dmz-low-priority
pf-dmz.conf:pass out on $dmz_if inet proto icmp from any to
icmp-type 11 code 0 queue dmz-low-priority
pf-dmz.conf:pass out on $dmz_if inet proto tcp from any to
port { 80 443 } queue dmz-low-priority
pf-ece.conf:pass out on $svr_if inet proto icmp from any to
icmp-type 8 code 0 queue svr-low-priority
pf-ece.conf:pass out on $svr_if inet proto icmp from any to
icmp-type 11 code 0 queue svr-low-priority
pf-ece.conf:pass out on $svr_if inet proto udp from any to
port 123 queue svr-low-priority
pf-ece.conf:pass out on $svr_if inet proto { tcp udp } from any to
port 53 queue svr-low-priority
pf-eng.conf:pass out on $svr_if inet proto icmp from any to
icmp-type 8 code 0 queue svr-low-priority
pf-eng.conf:pass out on $svr_if inet proto icmp from any to
icmp-type 11 code 0 queue svr-low-priority
pf-nix.conf:pass out on $svr_if inet proto tcp from any to
port 80 queue svr-low-priority
pf-nix.conf:pass out on $s
Vlan tagging and Carp
Hello all! I'm having some trouble with getting an OpenBSD box to properly tag packets via 802.1Q. I'm setting up an OpenBSD4.2 router pulling data off a trunk port on a Cisco 2960 switch. I can see the packets traverse the stack upwards but they are not being encapsulated with a vlan tag on the way back out so are getting dropped by the switch. Here's a diagram: - Other Machine arp who-has A.A.A.A - switch vlan20 (arp who-has A.A.A.A) - nfe0 vlan20 (arp who-has A.A.A.A) - trunk0 vlan20 (arp who-has A.A.A.A) - vlan20 arp who-has A.A.A.A - carp0 (A.A.A.A) arp who-has A.A.A.A - carp0 (A.A.A.A) arp reply 'I have A.A.A.A!' ... - nfe0 arp reply 'I have A.A.A.A!' - switch wtf?! nfe0/1, trunk0, vlan10/20/30 have no ip addresses. The carp0/1/2 sitting on the vlan pseudo-devices are the only devices with ip addresses. I was in a hurry yesterday so I didn't check to see if I was getting the same arp-replies on the vlan20 and trunk0 devices as I suppose it is possible that the packets were traversing another path down the network stack. I'll check tonight if I can. Any ideas or suggestions? G0kita
Re: Vlan tagging and Carp
Thanks Brian! I was really hoping to keep the number of ip addresses used to a minimum though. I'm using a /29 and am really tight on ips. Using carp0 straight onto (carpdev) nfe0 would not require nfe0 to have an ip address, any way I can get away with the same when the carpdev is vlan20? hostname.nfe0 up hostname.nfe1 up hostname.trunk0 trunkproto round-robin trunkport nfe0 trunkport nfe1 up hostname.vlan10 vlan 10 vlandev trunk0 hostname.vlan20 vlan 20 vlandev trunk0 hostname.carp0 inet 192.168.1.1 255.255.255.248 192.168.1.7 carpdev vlan10 vhid 0 advskew 0 pass password hostname.carp1 inet 192.168.1.9 255.255.255.248 192.168.1.15 carpdev vlan20 vhid 1 advskew 0 pass password On Wed, Mar 26, 2008 at 9:42 AM, Brian A. Seklecki < [EMAIL PROTECTED]> wrote: > > On Wed, 2008-03-26 at 09:32 -0400, G 0kita wrote: > > Hello all! I'm having some trouble with getting an OpenBSD box to > properly > > tag packets via 802.1Q. > > I'm setting up an OpenBSD4.2 router pulling data off a trunk port on a > Cisco > > 2960 switch. I can see the packets traverse the stack upwards but they > are > > The two physical/vlan interfaces on each unit should have an IP address > in the subnet (.2 and .3 respectively, normally). The CARP interface on > each system on each box should have the same address (.1 normally)
Re: Vlan tagging and Carp
Ok, managed to get back on the box for a little more troubleshooting! Interesting part is that there's no traffic on the carp pseudo interface although the machine certainly knows about it as it responds to the arp. I tried setting an ip address on the vlan30 interface but there was no change in the dump traffic behaviour. How come traffic leaving a vlan device isn't being encapsulated? Is the unencapsulated packet showing up on the trunk1 device through some other path? Any comments? Here's some dump traffic: Nothing from : carp3 carp2 carp1 vlan20 vlan10 vlan30 05:32:37.664607 arp who-has 1.1.1.189 tell 1.1.1.188 05:32:37.664621 arp reply 1.1.1.189 is-at 00:00:5e:00:01:04 trunk1 05:32:37.664603 802.1Q vid 30 pri 0 arp who-has 1.1.1.189 tell 1.1.1.188 05:32:37.664626 arp reply 1.1.1.189 is-at 00:00:5e:00:01:04 nfe1 05:32:37.664599 802.1Q vid 30 pri 0 arp who-has 1.1.1.189 tell 1.1.1.188 05:32:37.664631 arp reply 1.1.1.189 is-at 00:00:5e:00:01:04 The pertinent routing: DestinationGatewayFlagsRefs UseMtu Interface 1.1.1.184/29 1.1.1.189 U 1 539 - carp3 1.1.1.189 1.1.1.189 UH 02 - carp3 And some configuration files (again): /etc/hostname.carp1:inet 2.2.2.246 255.255.255.248 2.2.2.247 vhid 2 carpdev vlan10 pass password advbase 1 advskew 0 down /etc/hostname.carp2:inet 2.2.2.254 255.255.255.248 2.2.2.255 vhid 3 carpdev vlan20 pass password advbase 1 advskew 0 /etc/hostname.carp3:inet 1.1.1.189 255.255.255.248 1.1.1.191 vhid 4 carpdev vlan30 pass password advbase 1 advskew 0 /etc/hostname.nfe0:up /etc/hostname.nfe1:up /etc/hostname.trunk1:trunkproto failover trunkport nfe0 trunkport nfe1 up /etc/hostname.vlan10:vlan 10 vlandev trunk1 up /etc/hostname.vlan20:vlan 20 vlandev trunk1 up /etc/hostname.vlan30:vlan 30 vlandev trunk1 up I'm running a Generic kernel on an amd64 save for RAID being enabled.
NTP offline local server question
Hi all, I'm running a network simulation offline and wanted to have all the computers synchronized to a single time source. It's easy to set up the machines to query a central OpenNTP server but without an accurate time source that central OpenNTP server won't serve. The full ntp uses 127.127.1.0 as a local clock source and allows the machine to serve with only that as its source.Is there similar functionality or a dummy sensor device in /dev I can use? I understand that without drift information the time won't be accurate to the rest of the world but I really only want local synchronization. If not I can use the port easily enough, but I was wondering if there's a tweak I'm missing somewhere. G
mount_null replacement?
Hi there, I've been using null mounts for the last while to maintain a readonly filesystem under OpenBSD 3.6. For example: /etc/fstab: /home/user/dir /var/www/dir null ro,nodev,nosuid,noexec 0 0 I just tried this on an OpenBSD 3.9 system and it fails on the helper program for null (ie. mount_null). I notice mount_null was dropped as of OpenBSD 3.8, can someone tell me first of all why this was done (the changelog only shows comments about realpath failure). Secondly is there a replacement or workaround which will give me the kind of behaviour I'm looking for? Specifically I'm looking to have a writable directory mounted read-only in another location. Svnd mounts aren't the solution I'm looking for, I'm fairly sure. G.0kita
Re: Conundrum with aucat and rc_scripts
On Thu, Sep 8, 2011 at 10:01 PM, Breen Ouellette wrote: > I've configured the ices package to stream whatever happens to be flowing > into my sound card line input using this roundabout method (seems to work > the best given that ices will read from a FIFO but not stdin): > 1. aucat writes line in to FIFO at /dev/aucat/.raw; > 2. lame reads from above and writes to FIFO /dev/lame/.mp3; > 3. ices reads from above and sends to my icecast server. > > The following commands in a sh script run from root's shell form the meat > of the above chain of events: > > /usr/local/bin/lame --quiet -r -a -b 56 /dev/aucat/.raw /dev/lame/.mp3 & > /usr/bin/aucat -o - > /dev/aucat/.raw & > /etc/rc.d/ices start > > However, if I try to adjust /etc/rc.local to include the first two lines > (which need to be running before ices gets called by rc_scripts in > rc.conf.local), aucat refuses to start. > I've also taken the above commands and created a slightly more robust > watchdog script that is run as a cronjob. > > crontab entry: * * * * * /root/bin/wd_ices.sh > > /root/bin/wd_ices.sh: > snip > > > Unfortunately, this doesn't work exactly as expected either. While aucat > actually starts up, cron doesn't seem to like something about it and gets > stuck trying to send a message to root. `ps ax` shows the problem, which > just stalls there and won't go away: > > -PID- ?? I 0:00.04 /usr/sbin/sendmail -FCronDaemon -odi -oem -oi -t > > If I kill lame (which brings down aucat and ices), sendmail will then get > the message through and exit. > > Can anyone tell me how to get lame and aucat running properly at startup > before /etc/rc.d/ices gets called by rc.local? > > Can anyone tell me how to get the same working with cron without those > sendmail problems? > > Thanks. > > Breeno > > Hi Breeno, my suggestion is to use ezstream http://www.icecast.org/ezstream.php from the same icecast group instead. It will take a stream so you can use: aucat -o - | lame -r - - | ezstream -c ezstream.xml If you throw it into rc.local you will have to disconnect it from the terminal for it to work. /etc/rc.local: /usr/bin/aucat -o - 2> /dev/null | /usr/local/bin/lame -r - - | /usr/local/bin/ezstream -q -c /etc/ezstream.xml < /dev/null &> /dev/null or I've dumped it into a script aucat | lame | ezstream echo $! > stream.pid to use pid checking instead of that ps grepping (killing the ezstream will terminate the entire command set) and then you can also use an rc.d script daemon="script" daemon_user=_no_priv daemon_flags= /dev/null" Funny story, I'm working on something similar and will shortly have a question for misc too - watch for it as aucat is giving me some trouble. G0kita
