Hi,
I have an NTP server behind an OpenBSD firewall / router and seeing some
packet loss. The NTP server (Leontp 1200) should be able to handle the
load easily, so I suspect the packet loss occurs at the firewall/router
or elsewhere. So far I didn’t manage to find the cause, let alone solve
it. Any help is much appreciated.
My setup:
* 1Gbps Fiber connected to mediaconverter (fiber / ethernet)
* N100 system with 4 x igc(4), 8 GB ram, running 7.6-amd64 with
patches (dmesg below)
* igc0 connected to mediaconverter (using PPPoE over VLAN for
connecting to my ISP)
* igc1 connected to my local network
* igc2 connected to 8-port unmanaged 1Gbps switch, Leontp (with a 100
Mbps ethernet port) is connected to this switch with some other NTP
stuff
At the moment, NTP traffic is about 2k requests per second. The server
is part of the NTP pool and the pool monitoring system shows that there
is some packet loss. Not much, but enough to impact the overal score
and remove the server from the pool.
My first suspect was the state table on the firewall/router. Due to the
nature of the traffic, nearly every incoming packet creates a new state,
leading to an average of about 50k to 70k states. Close to the default
limit of 100k, so perhaps the limit was reached during peak traffic.
The ‘memory’ counter from pfctl -si was also at ~37k.
I increased the state limit to 500k and added ‘keep state (udp.single 5
udp.first 5 udp.multiple 5)’ to the rules that concern the NTP traffic.
The state table is now around 20 – 30k, well below the limit. The load
on the router is also quite low (~98% idle).
However, packet loss remains, albeit slightly lower.
My second thought was that the switch may be dropping packets during
peaks, as it has to buffer due to the difference in link speeds. The
Leontp has a 100 Mbps interface, while the rest is at 1 Gbps.
I added a 95M queue in my pf.conf with a qlimit of 1000, effectively
reducing the link speed between the OpenBSD and the switch to < 100
Mbps, but the packet loss remains.
pfctl -sq -vv shows that the NTP traffic is just over 1 Mbps without any
dropped packets.
My pf.conf is as follows:
# Macros
ext_if = "pppoe0"
lan_if = "igc1"
ntp_if = "igc2"
# Options
set limit states 500000
set block-policy return
set loginterface $ext_if
set skip on lo
queue std on $ntp_if bandwidth 100M
queue ntp parent std bandwidth 90M max 95M default qlimit 100
# Filter packets
block all
# antispoof rules
antispoof quick for { lo $ext_if }
# Translate packets
match out on $ext_if \
from 192.168.0.0/16 \
to any \
nat-to ($ext_if)
# ftp-proxy
anchor "ftp-proxy/*"
pass in quick on { $lan_if } \
inet proto tcp \
from 192.168.0.0/16 \
to port ftp \
divert-to 127.0.0.1 port 8021
# ntp
pass in quick on $ext_if inet proto udp \
from any to ($ext_if) port 123 rdr-to 192.168.4.123 \
keep state (udp.first 5 udp.multiple 5 udp.single 5)
# dhcp6c replies
pass in on $ext_if inet6 proto udp \
from fe80::/10 port 547 \
to port 546
# rest
pass in on { $lan_if $ntp_if }
# pass out rules
pass out on $ext_if
pass out on $ntp_if keep state (udp.first 5 udp.multiple 5 udp.single 5) \
set queue ntp
# traceroutes
pass proto udp \
to port 33433 >< 33626
# ICMP
pass inet proto icmp \
all icmp-type echoreq
pass inet6 proto icmp6 \
all icmp6-type { echoreq neighbrsol neighbradv }
pass on { $lan_if $ntp_if } inet6 proto icmp6 \
all icmp6-type { routersol routeradv }
I’m running out of ideas where to look next. Is there any way I can see
if/where packets are dropped?
Thanks in advance,
Maurice
OpenBSD 7.6 (GENERIC.MP) #338: Mon Sep 30 08:55:35 MDT 2024
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8304345088 (7919MB)
avail mem = 8029401088 (7657MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.6 @ 0x73c6f000 (119 entries)
bios0: vendor American Megatrends International, LLC. version
"HSX1264NPV10R006" date 11/07/2023
bios0: Default string Default string
efi0 at bios0: UEFI 2.8
efi0: American Megatrends rev 0x5001a
acpi0 at bios0: ACPI 6.4
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP FIDT SSDT SSDT SSDT SSDT HPET APIC MCFG SSDT
UEFI PSDS NHLT LPIT SSDT SSDT DBGP DBG2 DMAR FPDT SSDT SSDT SSDT SSDT
PHAT TPM2 WSMT
acpi0: wakeup devices PEGP(S4) PEGP(S4) PEGP(S4) SIO1(S3) RP09(S4)
PXSX(S4) RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) PXSX(S4) RP13(S4)
PXSX(S4) RP14(S4) PXSX(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 19200000 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) N100, 3392.19 MHz, 06-be-00, patch 0000001a
cpu0: cpuid 1
edx=bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
ecx=77fafbbf<SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND>
cpu0: cpuid 6 eax=578ff7<SENSOR,ARAT> ecx=9<EFFFREQ>
cpu0: cpuid 7.0
ebx=239ca7eb<FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PT,SHA>
ecx=98c007ac<UMIP,PKU,WAITPKG,PKS>
edx=fc184410<MD_CLEAR,IBT,IBRS,IBPB,STIBP,L1DF,SSBD>
cpu0: cpuid a vers=5, gp=6, gpwidth=48, ff=3, ffwidth=48
cpu0: cpuid d.1 eax=f<XSAVEOPT,XSAVEC,XGETBV1,XSAVES>
cpu0: cpuid 80000001 edx=2c100800<NXE,PAGE1GB,RDTSCP,LONG>
ecx=121<LAHF,ABM,3DNOWP>
cpu0: cpuid 80000007 edx=100<ITSC>
cpu0: msr
10a=1580fd6b<IBRS_ALL,SKIP_L1DFL,MDS_NO,IF_PSCHANGE,TAA_NO,MISC_PKG_CT,ENERGY_FILT,DOITM,SBDR_SSDP_N,FBSDP_NO,PSDP_NO,OVERCLOCK,PBRSB_NO,GDS_NO,RFDS_CLEAR>
cpu0: 32KB 64b/line 8-way D-cache, 64KB 64b/line 8-way I-cache, 2MB
64b/line 16-way L2 cache, 6MB 64b/line 12-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 38MHz
cpu0: mwait min=64, max=64, C-substates=0.2.0.2.0.1.0.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) N100, 3392.18 MHz, 06-be-00, patch 0000001a
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) N100, 3092.87 MHz, 06-be-00, patch 0000001a
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) N100, 2893.33 MHz, 06-be-00, patch 0000001a
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 120 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xc0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PC00)
acpiprt1 at acpi0: bus 4 (RP09)
acpiprt2 at acpi0: bus -1 (RP10)
acpiprt3 at acpi0: bus -1 (RP11)
acpiprt4 at acpi0: bus -1 (RP12)
acpiprt5 at acpi0: bus -1 (RP13)
acpiprt6 at acpi0: bus -1 (RP14)
acpiprt7 at acpi0: bus -1 (RP15)
acpiprt8 at acpi0: bus -1 (RP16)
acpiprt9 at acpi0: bus -1 (RP01)
acpiprt10 at acpi0: bus -1 (RP02)
acpiprt11 at acpi0: bus 1 (RP03)
acpiprt12 at acpi0: bus 2 (RP04)
acpiprt13 at acpi0: bus -1 (RP05)
acpiprt14 at acpi0: bus -1 (RP06)
acpiprt15 at acpi0: bus 3 (RP07)
acpiprt16 at acpi0: bus -1 (RP08)
acpiprt17 at acpi0: bus -1 (RP17)
acpiprt18 at acpi0: bus -1 (RP18)
acpiprt19 at acpi0: bus -1 (RP19)
acpiprt20 at acpi0: bus -1 (RP20)
acpiprt21 at acpi0: bus -1 (RP21)
acpiprt22 at acpi0: bus -1 (RP22)
acpiprt23 at acpi0: bus -1 (RP23)
acpiprt24 at acpi0: bus -1 (RP24)
acpiprt25 at acpi0: bus -1 (RP25)
acpiprt26 at acpi0: bus -1 (RP26)
acpiprt27 at acpi0: bus -1 (RP27)
acpiprt28 at acpi0: bus -1 (RP28)
acpiec0 at acpi0: not present
acpipci0 at acpi0 PC00: 0x00000000 0x00000011 0x00000001
com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com0: console
"ACPI000E" at acpi0 not configured
acpibtn0 at acpi0: SLPB
acpicpu0 at acpi0: C3(200@1048 mwait.1@0x60), C2(350@127 mwait.1@0x21),
C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C3(200@1048 mwait.1@0x60), C2(350@127 mwait.1@0x21),
C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C3(200@1048 mwait.1@0x60), C2(350@127 mwait.1@0x21),
C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C3(200@1048 mwait.1@0x60), C2(350@127 mwait.1@0x21),
C1(1000@1 mwait.1), PSS
"PNP0C14" at acpi0 not configured
"PNP0C14" at acpi0 not configured
intelpmc0 at acpi0: PEPD
state 0: 0x7f:1:2:0x00:0x0000000000000060
counter: 0x7f:64:0:0x00:0x0000000000000632
frequency: 0
state 1: 0x7f:1:2:0x00:0x0000000000000060
counter: 0x00:32:0:0x03:0x00000000fe00193c
frequency: 8197
acpibtn1 at acpi0: PWRB
tpm0 at acpi0 TPM_ 2.0 (CRB) addr 0xfed40000/0x5000, device 0x00000000
rev 0x0
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
acpipwrres0 at acpi0: WRST
acpipwrres1 at acpi0: FN00, resource for FAN0
acpipwrres2 at acpi0: FN01, resource for FAN1
acpipwrres3 at acpi0: FN02, resource for FAN2
acpipwrres4 at acpi0: FN03, resource for FAN3
acpipwrres5 at acpi0: FN04, resource for FAN4
acpitz0 at acpi0: critical temperature is 110 degC
acpipwrres6 at acpi0: PIN_
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD1F
acpivout1 at acpivideo0: DD2F
cpu0: using VERW MDS workaround
cpu0: Enhanced SpeedStep 3392 MHz: speeds: 801, 800, 700 MHz
pci0 at mainbus0 bus 0
0:31:5: mem address conflict 0xfe010000/0x1000
pchb0 at pci0 dev 0 function 0 "Intel N100 Host" rev 0x00
inteldrm0 at pci0 dev 2 function 0 "Intel Graphics" rev 0x00
drm0 at inteldrm0
inteldrm0: msi, ALDERLAKE_P, gen 12
"Intel Core 12G CL" rev 0x01 at pci0 dev 10 function 0 not configured
xhci0 at pci0 dev 20 function 0 "Intel ADL-N xHCI" rev 0x00: msi, xHCI 1.20
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev
3.00/1.00 addr 1
"Intel ADL-N SRAM" rev 0x00 at pci0 dev 20 function 2 not configured
"Intel ADL-N HECI" rev 0x00 at pci0 dev 22 function 0 not configured
ahci0 at pci0 dev 23 function 0 "Intel ADL-N AHCI" rev 0x00: msi, AHCI 1.3.1
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, Kingsand T600 12, V101>
t10.ATA_Kingsand_T600_128G_2024011300008909_
sd0: 122104MB, 512 bytes/sector, 250069680 sectors, thin
sdhc0 at pci0 dev 26 function 0 "Intel ADL-N eMMC" rev 0x00: apic 2 int 16
sdhc0: SDHC 3.00, 200 MHz base clock
sdmmc0 at sdhc0: 8-bit, sd high-speed, mmc high-speed, ddr52, dma
ppb0 at pci0 dev 28 function 0 "Intel ADL-N PCIE" rev 0x00: msi
pci1 at ppb0 bus 1
igc0 at pci1 dev 0 function 0 "Intel I226-V" rev 0x04, msix, 4 queues,
address 60:be:b4:14:cf:44
ppb1 at pci0 dev 28 function 3 "Intel ADL-N PCIE" rev 0x00: msi
pci2 at ppb1 bus 2
igc1 at pci2 dev 0 function 0 "Intel I226-V" rev 0x04, msix, 4 queues,
address 60:be:b4:14:cf:45
ppb2 at pci0 dev 28 function 6 "Intel ADL-N PCIE" rev 0x00: msi
pci3 at ppb2 bus 3
igc2 at pci3 dev 0 function 0 "Intel I226-V" rev 0x04, msix, 4 queues,
address 60:be:b4:14:cf:46
ppb3 at pci0 dev 29 function 0 "Intel ADL-N PCIE" rev 0x00: msi
pci4 at ppb3 bus 4
igc3 at pci4 dev 0 function 0 "Intel I226-V" rev 0x04, msix, 4 queues,
address 60:be:b4:14:cf:47
pcib0 at pci0 dev 31 function 0 "Intel ADL-N eSPI" rev 0x00
azalia0 at pci0 dev 31 function 3 "Intel ADL-N HD Audio" rev 0x00: msi
azalia0: no HD-Audio codecs
ichiic0 at pci0 dev 31 function 4 "Intel ADL-N SMBus" rev 0x00: apic 2
int 16
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 8GB DDR4 SDRAM PC4-19200 SO-DIMM
"Intel ADL-N SPI" rev 0x00 at pci0 dev 31 function 5 not configured
isa0 at pcib0
isadma0 at isa0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
efifb at mainbus0 not configured
sdmmc0: can't enable card
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (a1f99982312c6ed6.a) swap on sd0b dump on sd0b
inteldrm0: 1024x768, 32bpp
wsdisplay0 at inteldrm0 mux 1
wsdisplay0: screen 0-5 added (std, vt100 emulation)
