On Fri, Nov 24, 2006 at 10:22:26AM +0000, Brian Candler wrote:
> To answer my own question: inspired by the output of ipsecctl, I wrote a
> perl program (attached) to generate a suitable isakmpd.conf (also attached),
> and this appears to work just fine.
And now I seem to have hit some sort of scalability problem.
Generating 1,000 transport mode SAs, and monitoring them with
ipsecctl -s flow | grep wc -l
gives the following after isakmpd has been started:
Time(s) Num flows
------- ---------
10 606
20 976
30 1286
40 1384
50 1768
60 1946
70 1946 ..
And there it stops, never reaching 2000 (in+out).
But I find the following in /var/log/messages:
Nov 24 11:12:45 gw isakmpd[32720]: pf_key_v2_set_spi: UPDATE: No such process
Nov 24 11:12:45 gw last message repeated 26 times
1946 + 27*2 = 2000, so that's where the missing flows have gone. For some
reason some of them are not known; maybe some earlier messages to the kernel
were silently dropped?
A bit more background:
* The OpenBSD machine is a HP/Compaq desktop, single 2.8GHz processor,
512MB, rl0 interface
* The Cisco is a 7301 with VAM2+ crypto accelerator. It barely breaks
a sweat (peak CPU usage around 25% with all these SAs coming in)
* Connected via cheap 100M switch
I'm using isakmpd.conf as generated by the Perl script posted before,
setting up separate SAs for UDP ports 10000 to 10999 inclusive. I've also
added
[General]
Exchange-max-time=180
Retransmits=10
at the top.
OK, so next I tried
# isakmpd -c /etc/isakmpd/isakmpd.conf.1000 -K -4 -v -d -D 5=99 >log.out 2>&1
but that actually made the problem go away - all 2000 flows were set up
correctly :-( I think that the extra work of writing debug info slowed it
down sufficiently that whatever was overflowing before is not overflowing
now. About 16MB of logs were generated.
Next I tried less debugging, with
# isakmpd -c /etc/isakmpd/isakmpd.conf.1000 -K -4 -v -d -D 5=50 >log.out3 2>&1
With this the number of flows maxed out at 1840. The logs include things
like:
...
113517.306433 Sdep 50 pf_key_v2_get_spi: spi:
113517.306449 Sdep 50 856af6c7
...
113630.081939 Sdep 40 pf_key_v2_convert_id: IPv4 address 10.1.1.6/32
113630.081951 Sdep 40 pf_key_v2_convert_id: IPv4 address 10.1.1.1/32
113630.081966 Sdep 10 pf_key_v2_set_spi: satype 2 dst 10.1.1.6 SPI 0x856af6c7
113630.082048 Default pf_key_v2_set_spi: UPDATE: No such process
I can upload this whole log file if anyone wants to see it (~1MB
uncompressed). But perhaps an IPSEC guru can suggest some better way to pin
this down.
Finally, I thought I'd give it a go with 10,000 SAs, which is the sort of
scale I wanted to test the Cisco with anyway.
# isakmpd -c /etc/isakmpd/isakmpd.conf.10000 -K -4 -v -d -D 5=50 >log.out4 2>&1
It takes a few minutes for isakmpd to get going (although the OpenBSD box
remains responsive throughout). Its size grows to 133MB, after which it
starts to shrink, and then grow again.
The number of flows is low: after 5 minutes it shows
# ipsecctl -s flow | wc -l
492
# grep "No such process" log.out4 | wc -l
33
After 10 minutes:
# ipsecctl -s flow | wc -l
2568
After 20 minutes:
# ipsecctl -s flow | wc -l
2992
The machine isn't swapping, and remains responsive although isakmpd is using
100% CPU. But the rate of successful SA setups is much lower than it was
with 1,000.
Anyway, I think OpenBSD aquits itself pretty well, and I'm not too worried
about it being able to set up 10,000 SAs, but with 1,000 SAs I think it
would be worth trying to nail down the pf_key UPDATE problem.
Regards,
Brian Candler.
