> On 11 Aug 2023, at 21:08, Marko Cupać <marko.cu...@mimar.rs> wrote:
>
> Hi,
>
> I have star topology network where dozens of spokes communicate with
> other spokes through central hub over GRE tunnels protected with
> transport-mode ipsec.
>
> This worked great for years, but lately all the locations got bandwidth
> upgrade (spokes: 10Mbit -> 50Mbit, hub: 2x200Mbit -> 2x500Mbit), and I'm
> starting to experience problems.
>
> Spokes have APU4D4s, and my tests show they can push up to 30Mbit/s of
> ipsec bidirectionally. Hub has HPE DL360g9 with Xeon CPU E5-2623 v4 @
> 2.60GHz and bge NICs, and it seems it can push no more than 200Mbit/s
> of ipsec bidirectionally (I have no chance to test this thoroughly in a
> lab, but what I see in production indicate this strongly).
>
> Are there any commands I can run which would indicate ipsec traffic is
> being throttled due to hardware being underspecced? top shows CPU is
> more than 50% idle. netstat shows ~10000 Ierrs / Ifail (no Oerrs /
> Ifail) on interfaces that deal with ipsec for two months worth of
> uptime.
>
> Would replacing Xeon box with AMD EPYC 7262 likely result in an
> improvement? Should I go for some NICs other than bge? What hardware do
> I need at Hub location to accomodate ~400Mbit/s of ipsec
> bidirectionally?
>From recent experience it looks like IPsec, and the crypto processing in
>particular, still runs under the giant kernel lock. This means you're only
>going to go as fast as a single core can go, and you'll be particularly
>sensitive to contention on that lock. The things you can do Right Now(tm) are:
- upgrade to a system with the fastest single core performance you can afford
- upgrade to -current
the pf purge code has been taken out from under the big kernel lock. if you
have a lot of pf states, this will give more time to crypto.
- pick faster crypto algorithms
you might already be using the fastest, so maybe this wont help.
- terminate ipsec on multiple hosts
two kernels will be faster than one. however, this adds complexity to the
network, so not an obvious benefit.
- try wireguard?
if it's a single tunnel IP tunnel (ie, one gre(4), and not egre(4)) between the
hubs and spokes then wg might be simpler and faster. simpler because wg is less
layers than gre over ipsec, and faster cos it should be able to do crypto in
parallel.
in the future i'm sure the ipsec stack will improve, but it's hard work that
takes time.
dlg
>
> Thank you in advance,
>
>
> --
> Before enlightenment - chop wood, draw water.
> After enlightenment - chop wood, draw water.
>
> Marko Cupać
> https://www.mimar.rs/
>
