Scenario: 7.6-stable running on a gateway, connected to the internet via
pppoe0 over vlan7, several downstream /24 network segments. iked(8) is
serving several clients, running mostly Mac OS, with policies like this:
ikev2 "foo" esp \
from 192.168.100.1 to dynamic \
from 192.168.5.0/24 to dynamic \
from 192.168.50.0/24 to dynamic \
[...]
peer any \
srcid bar dstid foo \
config address 192.168.100.0/24
where we have vlanN interfaces carrying the 192.168.5.1/24 etc. etc. atop
an igc(4) interface and 192.168.100.1/24 is on lo1 for debugging purposes.
There is a relayd running on the gateway itself, with relays listening
on both the pppoe0 address and 192.168.5.1 (vlan5).
Mac OS sets the MTU on its ipsec0 interface to 1280, so the scrub rules
in pf.conf look like this:
[...]
match in on pppoe0 scrub (max-mss 1440, no-df, random-id, reassemble tcp)
match out on pppoe0 scrub (max-mss 1440)
match on enc0 all scrub (max-mss 1280)
[...]
ICMP unreachable is allowed, of course.
This works fine for all downstream hosts, with tcpdump showing
consistent packet sizes of 1356 on pppoe0. But max-mss seems to be
ignored for all connections to the gateway host itself, including the
ones to relayd at 192.168.5.1, resulting in heavy fragmentation:
X.X.X.X.ipsec-nat-t > Y.Y.Y.Y.64096: truncated-udp - 15108 bytes
missing!udpencap: esp spi 0xe5d2e7e0 seq 4597 len 1464 (frag 22065:1472@0+)
[tos 0x20]
11:46:46.432345 X.X.X.X > Y.Y.Y.Y: (frag 22065:1472@1472+) [tos 0x20]
11:46:46.432346 X.X.X.X > Y.Y.Y.Y: (frag 22065:1472@2944+) [tos 0x2
0]
11:46:46.432347 X.X.X.X > Y.Y.Y.Y: (frag 22065:1472@4416+) [tos 0x20]
11:46:46.432348 X.X.X.X > Y.Y.Y.Y: (frag 22065:1472@5888+) [tos 0x20]
11:46:46.432349 X.X.X.X > Y.Y.Y.Y: (frag 22065:1472@7360+) [tos 0x20]
11:46:46.432351 X.X.X.X > Y.Y.Y.Y: (frag 22065:1472@8832+) [tos 0x20]
11:46:46.432352 X.X.X.X > Y.Y.Y.Y: (frag 22065:1472@10304+) [tos 0x20]
11:46:46.432353 X.X.X.X > Y.Y.Y.Y: (frag 22065:1472@11776+) [tos 0x20]
11:46:46.432354 X.X.X.X > Y.Y.Y.Y: (frag 22065:1472@13248+) [tos 0x20]
11:46:46.432355 X.X.X.X > Y.Y.Y.Y: (frag 22065:1472@14720+) [tos 0x20]
11:46:46.432356 X.X.X.X > Y.Y.Y.Y: (frag 22065:388@16192) [tos 0x20]
For Mac OS, this means performance degradation (around 2-3MiB/s vs
12MiB/s) and sometimes dropped connections. The one OpenBSD client is
coping a better, interestingly.
Any hints?
OpenBSD 7.6 (GENERIC.MP) #338: Mon Sep 30 08:55:35 MDT 2024
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4094173184 (3904MB)
avail mem = 3946905600 (3764MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xeaa00 (56 entries)
bios0: vendor American Megatrends Inc. version "5.13" date 02/03/2023
bios0: Default string Default string
efi0 at bios0: UEFI 2.7
efi0: American Megatrends rev 0x5000d
acpi0 at bios0: ACPI 6.1
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP FPDT FIDT MCFG DBG2 DBGP HPET LPIT APIC NPKT SSDT SSDT
SSDT SSDT SSDT SSDT SSDT UEFI TPM2 DMAR WDAT WSMT
acpi0: wakeup devices HDAS(S3) XHC_(S4) XDCI(S4) RP01(S4) PXSX(S4) RP02(S4)
PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4) PXSX(S4)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 19200000 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Celeron(R) N4000 CPU @ 1.10GHz, 1096.97 MHz, 06-7a-01, patch
00000042
cpu0: cpuid 1
edx=bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
ecx=47f8ebbf<SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND>
cpu0: cpuid 6 eax=57<SENSOR,ARAT> ecx=1<EFFFREQ>
cpu0: cpuid 7.0
ebx=2294e287<FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SHA>
ecx=40400004<UMIP> edx=ac000400<MD_CLEAR,IBRS,IBPB,STIBP,SSBD>
cpu0: cpuid a vers=4, gp=4, gpwidth=48, ff=3, ffwidth=48
cpu0: cpuid d.1 eax=f<XSAVEOPT,XSAVEC,XGETBV1,XSAVES>
cpu0: cpuid 80000001 edx=2c100800<NXE,PAGE1GB,RDTSCP,LONG> ecx=101<LAHF,3DNOWP>
cpu0: cpuid 80000007 edx=100<ITSC>
cpu0: msr
10a=14000c6a<IBRS_ALL,SKIP_L1DFL,MDS_NO,IF_PSCHANGE,MISC_PKG_CT,ENERGY_FILT,GDS_NO,RFDS_CLEAR>
cpu0: MELTDOWN
cpu0: 24KB 64b/line 6-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 19MHz
cpu0: mwait min=64, max=64, C-substates=0.2.0.2.4.2.1.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Celeron(R) N4000 CPU @ 1.10GHz, 1096.97 MHz, 06-7a-01, patch
00000042
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 20, 120 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 2 (RP01)
acpiprt2 at acpi0: bus 3 (RP02)
acpiprt3 at acpi0: bus 1 (RP03)
acpiprt4 at acpi0: bus -1 (RP04)
acpiprt5 at acpi0: bus -1 (RP05)
acpiprt6 at acpi0: bus -1 (RP06)
acpiec0 at acpi0: not present
acpipci0 at acpi0 PCI0: 0x00000004 0x00000011 0x00000001
com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
acpicmos0 at acpi0
acpibtn0 at acpi0: PWRB
intelpmc0 at acpi0: PEPD
state 0: 0x7f:1:2:0x00:0x0000000000000060
counter: 0x7f:64:0:0x00:0x0000000000000632
frequency: 0
tpm0 at acpi0 TPM_ 2.0 (CRB) addr 0xfed40000/0x5000, device 0x00000000 rev 0x0
"PNP0C0B" at acpi0 not configured
acpipwrres0 at acpi0: DRST
acpipwrres1 at acpi0: DRST
acpipwrres2 at acpi0: DRST
acpipwrres3 at acpi0: DRST
acpipwrres4 at acpi0: DRST
acpipwrres5 at acpi0: DRST
acpipwrres6 at acpi0: WRST
acpicpu0 at acpi0: C3(10@150 mwait.1@0x60), C2(10@50 mwait.1@0x21), C1(1000@1
mwait.1@0x1), PSS
acpicpu1 at acpi0: C3(10@150 mwait.1@0x60), C2(10@50 mwait.1@0x21), C1(1000@1
mwait.1@0x1), PSS
acpipwrres7 at acpi0: FN00, resource for FAN0
acpitz0 at acpi0: critical temperature is 95 degC
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD1F
cpu0: using VERW MDS workaround
cpu0: Enhanced SpeedStep 1096 MHz: speeds: 1101, 1100, 1000, 900, 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Gemini Lake Host" rev 0x03
inteldrm0 at pci0 dev 2 function 0 "Intel UHD Graphics 600" rev 0x03
drm0 at inteldrm0
inteldrm0: msi, GEMINILAKE, gen 9
azalia0 at pci0 dev 14 function 0 "Intel Gemini Lake HD Audio" rev 0x03: msi
azalia0: no supported codecs
"Intel Gemini Lake MEI" rev 0x03 at pci0 dev 15 function 0 not configured
ahci0 at pci0 dev 18 function 0 "Intel Gemini Lake AHCI" rev 0x03: msi, AHCI
1.3.1
ahci0: PHY offline on port 0
ahci0: PHY offline on port 1
scsibus1 at ahci0: 32 targets
ppb0 at pci0 dev 19 function 0 "Intel Gemini Lake PCIE" rev 0xf3: msi
pci1 at ppb0 bus 1
nvme0 at pci1 dev 0 function 0 vendor "YMTC", unknown product 0x1013 rev 0x03:
msix, NVMe 1.3
nvme0: YMTC YMSS1ED02B21MC, firmware LN015216, serial YMA1128JA224350WW6
scsibus2 at nvme0: 2 targets, initiator 0
sd0 at scsibus2 targ 1 lun 0: <NVMe, YMTC YMSS1ED02B2, LN01>
sd0: 122104MB, 512 bytes/sector, 250069680 sectors
ppb1 at pci0 dev 20 function 0 "Intel Gemini Lake PCIE" rev 0xf3: msi
pci2 at ppb1 bus 2
igc0 at pci2 dev 0 function 0 "Intel I226-V" rev 0x04, msix, 2 queues, address
a8:b8:e0:03:74:bb
ppb2 at pci0 dev 20 function 1 "Intel Gemini Lake PCIE" rev 0xf3: msi
pci3 at ppb2 bus 3
igc1 at pci3 dev 0 function 0 "Intel I226-V" rev 0x04, msix, 2 queues, address
a8:b8:e0:03:74:bc
xhci0 at pci0 dev 21 function 0 "Intel Gemini Lake xHCI" rev 0x03: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00
addr 1
sdhc0 at pci0 dev 28 function 0 "Intel Gemini Lake eMMC" rev 0x03: apic 1 int 39
sdhc0: SDHC 3.00, 200 MHz base clock
sdmmc0 at sdhc0: 8-bit, sd high-speed, mmc high-speed, ddr52, dma
pcib0 at pci0 dev 31 function 0 "Intel Gemini Lake LPC" rev 0x03
ichiic0 at pci0 dev 31 function 1 "Intel Gemini Lake SMBus" rev 0x03: apic 1
int 20
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 4GB DDR4 SDRAM PC4-25600 SO-DIMM
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
efifb at mainbus0 not configured
sdmmc0: can't enable card
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (b263e417241586e8.a) swap on sd0b dump on sd0b
inteldrm0: 1024x768, 32bpp
wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation), using wskbd0
wsdisplay0: screen 1-5 added (std, vt100 emulation)
