On Tue, Dec 17, 2024 at 06:31:36PM +0100, Maurice Janssen wrote:
> Hi,
>
> I have an NTP server behind an OpenBSD firewall / router and seeing some
> packet loss.?? The NTP server (Leontp 1200) should be able to handle the
> load easily, so I suspect the packet loss occurs at the firewall/router or
> elsewhere.?? So far I didn???t manage to find the cause, let alone solve
> it.?? Any help is much appreciated.
>
> My setup:
>
> * 1Gbps Fiber connected to mediaconverter (fiber / ethernet)
> * N100 system with 4 x igc(4), 8 GB ram, running 7.6-amd64 with
> patches (dmesg below)
> * igc0 connected to mediaconverter (using PPPoE over VLAN for
> connecting to my ISP)
> * igc1 connected to my local network
> * igc2 connected to 8-port unmanaged 1Gbps switch, Leontp (with a 100
> Mbps ethernet port) is connected to this switch with some other NTP
> stuff
>
> At the moment, NTP traffic is about 2k requests per second.?? The server is
> part of the NTP pool and the pool monitoring system shows that there is some
> packet loss.?? Not much, but enough to impact the overal score and remove
> the server from the pool.
>
> My first suspect was the state table on the firewall/router.?? Due to the
> nature of the traffic, nearly every incoming packet creates a new state,
> leading to an average of about 50k to 70k states.?? Close to the default
> limit of 100k, so perhaps the limit was reached during peak traffic.?? The
> ???memory??? counter from pfctl -si was also at ~37k.
>
> I increased the state limit to 500k and added ???keep state (udp.single 5
> udp.first 5 udp.multiple 5)??? to the rules that concern the NTP traffic.??
> The state table is now around 20 ??? 30k, well below the limit.?? The load
> on the router is also quite low (~98% idle).
>
> However, packet loss remains, albeit slightly lower.
>
> My second thought was that the switch may be dropping packets during peaks,
> as it has to buffer due to the difference in link speeds.?? The Leontp has a
> 100 Mbps interface, while the rest is at 1 Gbps.
>
> I added a 95M queue in my pf.conf with a qlimit of 1000, effectively
> reducing the link speed between the OpenBSD and the switch to < 100 Mbps,
> but the packet loss remains.
>
> pfctl -sq -vv shows that the NTP traffic is just over 1 Mbps without any
> dropped packets.
>
>
> My pf.conf is as follows:
>
> # Macros
> ext_if???????? = "pppoe0"
> lan_if???????? = "igc1"
> ntp_if???????? = "igc2"
>
>
> # Options
> set limit states 500000
> set block-policy return
> set loginterface $ext_if
> set skip on lo
>
> queue std on $ntp_if bandwidth 100M
> queue ntp parent std bandwidth 90M max 95M default qlimit 100
>
> # Filter packets
> block all
>
> # antispoof rules
> antispoof quick for { lo $ext_if }
>
> # Translate packets
> match out on $ext_if \
> ?????????????? from 192.168.0.0/16 \
> ?????????????? to any \
> ?????????????? nat-to ($ext_if)
>
> # ftp-proxy
> anchor "ftp-proxy/*"
> pass in quick on { $lan_if } \
> ?????????????? inet proto tcp \
> ?????????????? from 192.168.0.0/16 \
> ?????????????? to port ftp \
> ?????????????? divert-to 127.0.0.1 port 8021
>
> # ntp
> pass in quick on $ext_if inet proto udp \
> ?????????????? from any to ($ext_if) port 123 rdr-to 192.168.4.123 \
> ?????????????? keep state (udp.first 5 udp.multiple 5 udp.single 5)
>
> # dhcp6c replies
> pass in on $ext_if inet6 proto udp \
> ?????????????? from fe80::/10 port 547 \
> ?????????????? to port 546
>
> # rest
> pass in on { $lan_if $ntp_if }
>
> # pass out rules
> pass out on $ext_if
> pass out on $ntp_if keep state (udp.first 5 udp.multiple 5 udp.single 5) \
> ?????????????? set queue ntp
>
> # traceroutes
> pass proto udp \
> ?????????????? to port 33433 >< 33626
>
> # ICMP
> pass inet proto icmp \
> ?????????????? all icmp-type echoreq
> pass inet6 proto icmp6 \
> ?????????????? all icmp6-type { echoreq neighbrsol neighbradv }
> pass on { $lan_if $ntp_if } inet6 proto icmp6 \
> ?????????????? all icmp6-type { routersol routeradv }
>
> I???m running out of ideas where to look next.?? Is there any way I can see
> if/where packets are dropped?
Hey Maurice,
bluhm@ just did a talk at eurobsdcon that might help you understand the
different points to look at. my tldr version is if packets are being
dropped on the firewall it will be somewhere in the receive path. my
experience is that once a packet has made it into the network stack
it tends to get transmitted fine.
the places that packets can get dropped are in:
1. hardware
hardware counters from igc should be visible using kstat(1) now. the
igc-stats that you're looking at would be any of the errs/colls, or the
rx no bufs counter.
2. rxq drops and errors
the rxq is where packets received by the driver are put before being
handed over to the network stack. again, these are visible as kstats,
but are also aggregated as interface level counters you can see with
netstat -i. you can look at drops and errors individiually with netstat
-id and netstat -ie respectively.
the fact that you're using pppoe on top of a vlan adds some potential
for loss too. there's a queue in between ethernet input processing and
pppoe processing that could be dropping packets. i don't think we have a
way to see drops on that queue though.
if you're feeling brave you can try the diff i sent to the 'PPPoE
passthrough with "GigaHub" is very slow' thread on misc@ a week or so
ago which skips the queue for pppoe data packets.
3. pf drops
i think systat pf and pfctl -si are the best way to see counters here.
however, i feel like there's some things that pf does and doesn't do
sometimes that aren't captured well.
4. tx drops
again, i think this is unlikely, but you should be able to see tx
interface drops using the txq kstats or drops/errors in netstat -i
output. there's tx counters in the igc-stats too.
>
>
> Thanks in advance,
>
> Maurice
>
>
> OpenBSD 7.6 (GENERIC.MP) #338: Mon Sep 30 08:55:35 MDT 2024
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 8304345088 (7919MB)
> avail mem = 8029401088 (7657MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 3.6 @ 0x73c6f000 (119 entries)
> bios0: vendor American Megatrends International, LLC. version
> "HSX1264NPV10R006" date 11/07/2023
> bios0: Default string Default string
> efi0 at bios0: UEFI 2.8
> efi0: American Megatrends rev 0x5001a
> acpi0 at bios0: ACPI 6.4
> acpi0: sleep states S0 S3 S4 S5
> acpi0: tables DSDT FACP FIDT SSDT SSDT SSDT SSDT HPET APIC MCFG SSDT UEFI
> PSDS NHLT LPIT SSDT SSDT DBGP DBG2 DMAR FPDT SSDT SSDT SSDT SSDT PHAT TPM2
> WSMT
> acpi0: wakeup devices PEGP(S4) PEGP(S4) PEGP(S4) SIO1(S3) RP09(S4) PXSX(S4)
> RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) PXSX(S4) RP13(S4) PXSX(S4)
> RP14(S4) PXSX(S4) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpihpet0 at acpi0: 19200000 Hz
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) N100, 3392.19 MHz, 06-be-00, patch 0000001a
> cpu0: cpuid 1
> edx=bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>
> ecx=77fafbbf<SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND>
> cpu0: cpuid 6 eax=578ff7<SENSOR,ARAT> ecx=9<EFFFREQ>
> cpu0: cpuid 7.0
> ebx=239ca7eb<FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PT,SHA>
> ecx=98c007ac<UMIP,PKU,WAITPKG,PKS>
> edx=fc184410<MD_CLEAR,IBT,IBRS,IBPB,STIBP,L1DF,SSBD>
> cpu0: cpuid a vers=5, gp=6, gpwidth=48, ff=3, ffwidth=48
> cpu0: cpuid d.1 eax=f<XSAVEOPT,XSAVEC,XGETBV1,XSAVES>
> cpu0: cpuid 80000001 edx=2c100800<NXE,PAGE1GB,RDTSCP,LONG>
> ecx=121<LAHF,ABM,3DNOWP>
> cpu0: cpuid 80000007 edx=100<ITSC>
> cpu0: msr
> 10a=1580fd6b<IBRS_ALL,SKIP_L1DFL,MDS_NO,IF_PSCHANGE,TAA_NO,MISC_PKG_CT,ENERGY_FILT,DOITM,SBDR_SSDP_N,FBSDP_NO,PSDP_NO,OVERCLOCK,PBRSB_NO,GDS_NO,RFDS_CLEAR>
> cpu0: 32KB 64b/line 8-way D-cache, 64KB 64b/line 8-way I-cache, 2MB 64b/line
> 16-way L2 cache, 6MB 64b/line 12-way L3 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 38MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.0.2.0.1.0.1, IBE
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) N100, 3392.18 MHz, 06-be-00, patch 0000001a
> cpu1: smt 0, core 1, package 0
> cpu2 at mainbus0: apid 4 (application processor)
> cpu2: Intel(R) N100, 3092.87 MHz, 06-be-00, patch 0000001a
> cpu2: smt 0, core 2, package 0
> cpu3 at mainbus0: apid 6 (application processor)
> cpu3: Intel(R) N100, 2893.33 MHz, 06-be-00, patch 0000001a
> cpu3: smt 0, core 3, package 0
> ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 120 pins
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xc0000000, bus 0-255
> acpiprt0 at acpi0: bus 0 (PC00)
> acpiprt1 at acpi0: bus 4 (RP09)
> acpiprt2 at acpi0: bus -1 (RP10)
> acpiprt3 at acpi0: bus -1 (RP11)
> acpiprt4 at acpi0: bus -1 (RP12)
> acpiprt5 at acpi0: bus -1 (RP13)
> acpiprt6 at acpi0: bus -1 (RP14)
> acpiprt7 at acpi0: bus -1 (RP15)
> acpiprt8 at acpi0: bus -1 (RP16)
> acpiprt9 at acpi0: bus -1 (RP01)
> acpiprt10 at acpi0: bus -1 (RP02)
> acpiprt11 at acpi0: bus 1 (RP03)
> acpiprt12 at acpi0: bus 2 (RP04)
> acpiprt13 at acpi0: bus -1 (RP05)
> acpiprt14 at acpi0: bus -1 (RP06)
> acpiprt15 at acpi0: bus 3 (RP07)
> acpiprt16 at acpi0: bus -1 (RP08)
> acpiprt17 at acpi0: bus -1 (RP17)
> acpiprt18 at acpi0: bus -1 (RP18)
> acpiprt19 at acpi0: bus -1 (RP19)
> acpiprt20 at acpi0: bus -1 (RP20)
> acpiprt21 at acpi0: bus -1 (RP21)
> acpiprt22 at acpi0: bus -1 (RP22)
> acpiprt23 at acpi0: bus -1 (RP23)
> acpiprt24 at acpi0: bus -1 (RP24)
> acpiprt25 at acpi0: bus -1 (RP25)
> acpiprt26 at acpi0: bus -1 (RP26)
> acpiprt27 at acpi0: bus -1 (RP27)
> acpiprt28 at acpi0: bus -1 (RP28)
> acpiec0 at acpi0: not present
> acpipci0 at acpi0 PC00: 0x00000000 0x00000011 0x00000001
> com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
> com0: console
> "ACPI000E" at acpi0 not configured
> acpibtn0 at acpi0: SLPB
> acpicpu0 at acpi0: C3(200@1048 mwait.1@0x60), C2(350@127 mwait.1@0x21),
> C1(1000@1 mwait.1), PSS
> acpicpu1 at acpi0: C3(200@1048 mwait.1@0x60), C2(350@127 mwait.1@0x21),
> C1(1000@1 mwait.1), PSS
> acpicpu2 at acpi0: C3(200@1048 mwait.1@0x60), C2(350@127 mwait.1@0x21),
> C1(1000@1 mwait.1), PSS
> acpicpu3 at acpi0: C3(200@1048 mwait.1@0x60), C2(350@127 mwait.1@0x21),
> C1(1000@1 mwait.1), PSS
> "PNP0C14" at acpi0 not configured
> "PNP0C14" at acpi0 not configured
> intelpmc0 at acpi0: PEPD
> state 0: 0x7f:1:2:0x00:0x0000000000000060
> counter: 0x7f:64:0:0x00:0x0000000000000632
> frequency: 0
> state 1: 0x7f:1:2:0x00:0x0000000000000060
> counter: 0x00:32:0:0x03:0x00000000fe00193c
> frequency: 8197
> acpibtn1 at acpi0: PWRB
> tpm0 at acpi0 TPM_ 2.0 (CRB) addr 0xfed40000/0x5000, device 0x00000000 rev
> 0x0
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> acpipwrres0 at acpi0: WRST
> acpipwrres1 at acpi0: FN00, resource for FAN0
> acpipwrres2 at acpi0: FN01, resource for FAN1
> acpipwrres3 at acpi0: FN02, resource for FAN2
> acpipwrres4 at acpi0: FN03, resource for FAN3
> acpipwrres5 at acpi0: FN04, resource for FAN4
> acpitz0 at acpi0: critical temperature is 110 degC
> acpipwrres6 at acpi0: PIN_
> acpivideo0 at acpi0: GFX0
> acpivout0 at acpivideo0: DD1F
> acpivout1 at acpivideo0: DD2F
> cpu0: using VERW MDS workaround
> cpu0: Enhanced SpeedStep 3392 MHz: speeds: 801, 800, 700 MHz
> pci0 at mainbus0 bus 0
> 0:31:5: mem address conflict 0xfe010000/0x1000
> pchb0 at pci0 dev 0 function 0 "Intel N100 Host" rev 0x00
> inteldrm0 at pci0 dev 2 function 0 "Intel Graphics" rev 0x00
> drm0 at inteldrm0
> inteldrm0: msi, ALDERLAKE_P, gen 12
> "Intel Core 12G CL" rev 0x01 at pci0 dev 10 function 0 not configured
> xhci0 at pci0 dev 20 function 0 "Intel ADL-N xHCI" rev 0x00: msi, xHCI 1.20
> usb0 at xhci0: USB revision 3.0
> uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev
> 3.00/1.00 addr 1
> "Intel ADL-N SRAM" rev 0x00 at pci0 dev 20 function 2 not configured
> "Intel ADL-N HECI" rev 0x00 at pci0 dev 22 function 0 not configured
> ahci0 at pci0 dev 23 function 0 "Intel ADL-N AHCI" rev 0x00: msi, AHCI 1.3.1
> ahci0: port 0: 6.0Gb/s
> scsibus1 at ahci0: 32 targets
> sd0 at scsibus1 targ 0 lun 0: <ATA, Kingsand T600 12, V101>
> t10.ATA_Kingsand_T600_128G_2024011300008909_
> sd0: 122104MB, 512 bytes/sector, 250069680 sectors, thin
> sdhc0 at pci0 dev 26 function 0 "Intel ADL-N eMMC" rev 0x00: apic 2 int 16
> sdhc0: SDHC 3.00, 200 MHz base clock
> sdmmc0 at sdhc0: 8-bit, sd high-speed, mmc high-speed, ddr52, dma
> ppb0 at pci0 dev 28 function 0 "Intel ADL-N PCIE" rev 0x00: msi
> pci1 at ppb0 bus 1
> igc0 at pci1 dev 0 function 0 "Intel I226-V" rev 0x04, msix, 4 queues,
> address 60:be:b4:14:cf:44
> ppb1 at pci0 dev 28 function 3 "Intel ADL-N PCIE" rev 0x00: msi
> pci2 at ppb1 bus 2
> igc1 at pci2 dev 0 function 0 "Intel I226-V" rev 0x04, msix, 4 queues,
> address 60:be:b4:14:cf:45
> ppb2 at pci0 dev 28 function 6 "Intel ADL-N PCIE" rev 0x00: msi
> pci3 at ppb2 bus 3
> igc2 at pci3 dev 0 function 0 "Intel I226-V" rev 0x04, msix, 4 queues,
> address 60:be:b4:14:cf:46
> ppb3 at pci0 dev 29 function 0 "Intel ADL-N PCIE" rev 0x00: msi
> pci4 at ppb3 bus 4
> igc3 at pci4 dev 0 function 0 "Intel I226-V" rev 0x04, msix, 4 queues,
> address 60:be:b4:14:cf:47
> pcib0 at pci0 dev 31 function 0 "Intel ADL-N eSPI" rev 0x00
> azalia0 at pci0 dev 31 function 3 "Intel ADL-N HD Audio" rev 0x00: msi
> azalia0: no HD-Audio codecs
> ichiic0 at pci0 dev 31 function 4 "Intel ADL-N SMBus" rev 0x00: apic 2 int
> 16
> iic0 at ichiic0
> spdmem0 at iic0 addr 0x50: 8GB DDR4 SDRAM PC4-19200 SO-DIMM
> "Intel ADL-N SPI" rev 0x00 at pci0 dev 31 function 5 not configured
> isa0 at pcib0
> isadma0 at isa0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> vmm0 at mainbus0: VMX/EPT
> efifb at mainbus0 not configured
> sdmmc0: can't enable card
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> root on sd0a (a1f99982312c6ed6.a) swap on sd0b dump on sd0b
> inteldrm0: 1024x768, 32bpp
> wsdisplay0 at inteldrm0 mux 1
> wsdisplay0: screen 0-5 added (std, vt100 emulation)
>
>
