Hello Patrick, > In my opinion your net5501’s system calls per interval are relatively high. > The (traps sys) column on my firewall hovers between 40 & 50 quite > consistently. > My understanding is that system calls are things like program calls & library > access. Is there any way to decrease these values? > Many commercial routers run a customized kernel & rely on a striped down > user-land. > The kernel is also recompiled to run TCP/IP4 only & can no longer execute > things like storage or virtualization. > The OpenBSD O.S includes all the user-land tools such as ping & top in > addition to a standardized precompiled kernel. Ok, I get it.
On Fri, 23 Aug 2019 21:12:35 -0500 Patrick Dohman <dohmanpatr...@gmail.com> wrote: > In my opinion your net5501’s system calls per interval are relatively high. > The (traps sys) column on my firewall hovers between 40 & 50 quite > consistently. > My understanding is that system calls are things like program calls & library > access. > > In addition your net5501’s memory requests per second seem heavy. > You have fifty eight million 1024 bucket requests per second. > My firewall has a max of one hundred thousand 128 bucket requests per second. > > Many commercial routers run a customized kernel & rely on a striped down > user-land. > The kernel is also recompiled to run TCP/IP4 only & can no longer execute > things like storage or virtualization. > The OpenBSD O.S includes all the user-land tools such as ping & top in > addition to a standardized precompiled kernel. > Regards > Patrick > . > > > > > > On Thu, 22 Aug 2019 19:12:55 -0500 > > Patrick Dohman <dohmanpatr...@gmail.com> wrote: > > > >> Radek > >> > >> I’ve found that fast networking is actually CPU & memory intensive. > >> Pentium 4 and Xeon's are increasingly a necessity for stable firewalls in > >> my opinion. > >> Keep in mind OpenBSD is a monolithic kernel & isn’t a one to one ratio > >> with a commercial router. > >> > >> What are your context switches & interrupts doing while the VPN is up & > >> traffic is flowing? > >> > >> vmstat -w 4 > >> > >> What is your memory high water mark during a peak traffic? > >> > >> vmstat -m > >> > >> Regards > >> Patrick > >> > >>> On Aug 21, 2019, at 12:34 AM, radek <r...@int.pl> wrote: > >>> > >>> Hello Patrick, > >>> I am sorry for the late reply. > >>> > >>>> Do you consider memory an issue? > >>> No, I do not. I have a bunch of old Soekris/net5501-70 and ALIX2d2/2d3, > >>> that I use for VPN testing. > >>> Current testing set (6.5/i386) is net5501-70 <-> ALIX2d3 > >>> Production set (6.3/i386) is net5501-70 <-> ALIX2d2 > >>> Also have tried net5501-70 <-> net5501-70 - the same VPN problem occurs > >>> It is unlikely that every box has any hardware issue. > >>> > >>>> Unix load average can occasionally be deceiving. > >>> I did not know. > >>> > >>> #### net5501-70 #### > >>> $top -d1 | head -n 4 > >>> load averages: 0.05, 0.01, 0.00 RAC-fw65-test.PRAC 10:58:14 > >>> 38 processes: 1 running, 35 idle, 1 dead, 1 on processor up 3 days, 18:02 > >>> CPU states: 0.5% user, 0.0% nice, 0.4% sys, 0.0% spin, 0.2% intr, > >>> 98.8% idle > >>> Memory: Real: 18M/267M act/tot Free: 222M Cache: 97M Swap: 0K/256M > >>> > >>> #### ALIX2d3 #### > >>> $top -d1 | head -n 4 > >>> load averages: 0.00, 0.00, 0.00 mon65.home 07:30:05 > >>> 37 processes: 1 running, 35 idle, 1 on processor up 13:46 > >>> CPU states: 0.3% user, 0.0% nice, 1.1% sys, 0.0% spin, 0.4% intr, > >>> 98.3% idle > >>> Memory: Real: 125M/223M act/tot Free: 14M Cache: 47M Swap: 73M/256M > >>> > >>> > >>> > >>>> What is the speed of your memory? > >>>> What make of Ethernets are you running? > >>> Dmesgs below > >>> > >>> #### net5501-70 #### > >>> OpenBSD 6.5 (GENERIC) #2: Tue Jul 23 23:08:46 CEST 2019 > >>> r...@syspatch-65-i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC > >>> real mem = 536363008 (511MB) > >>> avail mem = 511311872 (487MB) > >>> mpath0 at root > >>> scsibus0 at mpath0: 256 targets > >>> mainbus0 at root > >>> bios0 at mainbus0: date 20/80/26, BIOS32 rev. 0 @ 0xfac40 > >>> pcibios0 at bios0: rev 2.0 @ 0xf0000/0x10000 > >>> pcibios0: pcibios_get_intr_routing - function not supported > >>> pcibios0: PCI IRQ Routing information unavailable. > >>> pcibios0: PCI bus #0 is the last bus > >>> bios0: ROM list: 0xc8000/0xa800 > >>> cpu0 at mainbus0: (uniprocessor) > >>> cpu0: Geode(TM) Integrated Processor by AMD PCS ("AuthenticAMD" > >>> 586-class) 500 MHz, 05-0a-02 > >>> cpu0: FPU,DE,PSE,TSC,MSR,CX8,SEP,PGE,CMOV,CFLUSH,MMX,MMXX,3DNOW2,3DNOW > >>> mtrr: K6-family MTRR support (2 registers) > >>> amdmsr0 at mainbus0 > >>> pci0 at mainbus0 bus 0: configuration mode 1 (bios) > >>> 0:20:0: io address conflict 0x6100/0x100 > >>> 0:20:0: io address conflict 0x6200/0x200 > >>> pchb0 at pci0 dev 1 function 0 "AMD Geode LX" rev 0x33 > >>> glxsb0 at pci0 dev 1 function 2 "AMD Geode LX Crypto" rev 0x00: RNG AES > >>> vr0 at pci0 dev 6 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 11, > >>> address 00:00:24:cb:4f:cc > >>> ukphy0 at vr0 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI > >>> 0x004063, model 0x0034 > >>> vr1 at pci0 dev 7 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 5, > >>> address 00:00:24:cb:4f:cd > >>> ukphy1 at vr1 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI > >>> 0x004063, model 0x0034 > >>> vr2 at pci0 dev 8 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 9, > >>> address 00:00:24:cb:4f:ce > >>> ukphy2 at vr2 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI > >>> 0x004063, model 0x0034 > >>> vr3 at pci0 dev 9 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 12, > >>> address 00:00:24:cb:4f:cf > >>> ukphy3 at vr3 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI > >>> 0x004063, model 0x0034 > >>> glxpcib0 at pci0 dev 20 function 0 "AMD CS5536 ISA" rev 0x03: rev 3, > >>> 32-bit 3579545Hz timer, watchdog, gpio, i2c > >>> gpio0 at glxpcib0: 32 pins > >>> iic0 at glxpcib0 > >>> pciide0 at pci0 dev 20 function 2 "AMD CS5536 IDE" rev 0x01: DMA, channel > >>> 0 wired to compatibility, channel 1 wired to compatibility > >>> wd0 at pciide0 channel 0 drive 0: <SanDisk SDCFH-008G> > >>> wd0: 1-sector PIO, LBA48, 7629MB, 15625216 sectors > >>> wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 > >>> pciide0: channel 1 ignored (disabled) > >>> ohci0 at pci0 dev 21 function 0 "AMD CS5536 USB" rev 0x02: irq 15, > >>> version 1.0, legacy support > >>> ehci0 at pci0 dev 21 function 1 "AMD CS5536 USB" rev 0x02: irq 15 > >>> usb0 at ehci0: USB revision 2.0 > >>> uhub0 at usb0 configuration 1 interface 0 "AMD EHCI root hub" rev > >>> 2.00/1.00 addr 1 > >>> isa0 at glxpcib0 > >>> isadma0 at isa0 > >>> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > >>> com0: console > >>> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo > >>> pckbc0 at isa0 port 0x60/5 irq 1 irq 12 > >>> pckbc0: unable to establish interrupt for irq 12 > >>> pckbd0 at pckbc0 (kbd slot) > >>> wskbd0 at pckbd0: console keyboard > >>> pcppi0 at isa0 port 0x61 > >>> spkr0 at pcppi0 > >>> nsclpcsio0 at isa0 port 0x2e/2: NSC PC87366 rev 9: GPIO VLM TMS > >>> gpio1 at nsclpcsio0: 29 pins > >>> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 > >>> usb1 at ohci0: USB revision 1.0 > >>> uhub1 at usb1 configuration 1 interface 0 "AMD OHCI root hub" rev > >>> 1.00/1.00 addr 1 > >>> vscsi0 at root > >>> scsibus1 at vscsi0: 256 targets > >>> softraid0 at root > >>> scsibus2 at softraid0: 256 targets > >>> root on wd0a (2bf8b7abbbce37df.a) swap on wd0b dump on wd0b > >>> > >>> > >>> #### ALIX2d3 #### > >>> OpenBSD 6.5 (GENERIC) #2: Tue Jul 23 23:08:46 CEST 2019 > >>> r...@syspatch-65-i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC > >>> real mem = 267931648 (255MB) > >>> avail mem = 247779328 (236MB) > >>> mpath0 at root > >>> scsibus0 at mpath0: 256 targets > >>> mainbus0 at root > >>> bios0 at mainbus0: date 11/05/08, BIOS32 rev. 0 @ 0xfd088 > >>> pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000 > >>> pcibios0: pcibios_get_intr_routing - function not supported > >>> pcibios0: PCI IRQ Routing information unavailable. > >>> pcibios0: PCI bus #0 is the last bus > >>> bios0: ROM list: 0xe0000/0xa800 > >>> cpu0 at mainbus0: (uniprocessor) > >>> cpu0: Geode(TM) Integrated Processor by AMD PCS ("AuthenticAMD" > >>> 586-class) 499 MHz, 05-0a-02 > >>> cpu0: FPU,DE,PSE,TSC,MSR,CX8,SEP,PGE,CMOV,CFLUSH,MMX,MMXX,3DNOW2,3DNOW > >>> mtrr: K6-family MTRR support (2 registers) > >>> pci0 at mainbus0 bus 0: configuration mode 1 (bios) > >>> pchb0 at pci0 dev 1 function 0 "AMD Geode LX" rev 0x33 > >>> glxsb0 at pci0 dev 1 function 2 "AMD Geode LX Crypto" rev 0x00: RNG AES > >>> vr0 at pci0 dev 9 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 10, > >>> address 00:0d:b9:1e:85:8c > >>> ukphy0 at vr0 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI > >>> 0x004063, model 0x0034 > >>> vr1 at pci0 dev 10 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 11, > >>> address 00:0d:b9:1e:85:8d > >>> ukphy1 at vr1 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI > >>> 0x004063, model 0x0034 > >>> vr2 at pci0 dev 11 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 15, > >>> address 00:0d:b9:1e:85:8e > >>> ukphy2 at vr2 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI > >>> 0x004063, model 0x0034 > >>> glxpcib0 at pci0 dev 15 function 0 "AMD CS5536 ISA" rev 0x03: rev 3, > >>> 32-bit 3579545Hz timer, watchdog, gpio, i2c > >>> gpio0 at glxpcib0: 32 pins > >>> iic0 at glxpcib0 > >>> maxtmp0 at iic0 addr 0x4c: lm86 > >>> pciide0 at pci0 dev 15 function 2 "AMD CS5536 IDE" rev 0x01: DMA, channel > >>> 0 wired to compatibility, channel 1 wired to compatibility > >>> wd0 at pciide0 channel 0 drive 0: <SanDisk SDCFH-008G> > >>> wd0: 1-sector PIO, LBA48, 7629MB, 15625216 sectors > >>> wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 > >>> pciide0: channel 1 ignored (disabled) > >>> ohci0 at pci0 dev 15 function 4 "AMD CS5536 USB" rev 0x02: irq 12, > >>> version 1.0, legacy support > >>> ehci0 at pci0 dev 15 function 5 "AMD CS5536 USB" rev 0x02: irq 12 > >>> usb0 at ehci0: USB revision 2.0 > >>> uhub0 at usb0 configuration 1 interface 0 "AMD EHCI root hub" rev > >>> 2.00/1.00 addr 1 > >>> isa0 at glxpcib0 > >>> isadma0 at isa0 > >>> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > >>> com0: console > >>> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo > >>> pcppi0 at isa0 port 0x61 > >>> spkr0 at pcppi0 > >>> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 > >>> usb1 at ohci0: USB revision 1.0 > >>> uhub1 at usb1 configuration 1 interface 0 "AMD OHCI root hub" rev > >>> 1.00/1.00 addr 1 > >>> nvram: invalid checksum > >>> vscsi0 at root > >>> scsibus1 at vscsi0: 256 targets > >>> softraid0 at root > >>> scsibus2 at softraid0: 256 targets > >>> root on wd0a (83b335c3c86bb80c.a) swap on wd0b dump on wd0b > >>> clock: unknown CMOS layout > >>> > >>> On Mon, 19 Aug 2019 18:17:48 -0500 > >>> Patrick Dohman <dohmanpatr...@gmail.com> wrote: > >>> > >>>> Do you consider memory an issue? > >>>> What is the speed of your memory? > >>>> Unix load average can occasionally be deceiving. > >>>> What make of Ethernets are you running? > >>>> Regards > >>>> Patrick > >>>> > >>>>> On Aug 19, 2019, at 5:28 AM, radek <r...@int.pl> wrote: > >>>>> > >>>>> Hello Patrick, > >>>>> > >>>>>> Does your ISP implement authoritative DNS? > >>>>>> Do you suspect a UDP issue? > >>>>> My VPN is configured with IPs, not with domain names. Does DNS and/or > >>>>> UDP matter anyway? > >>>>> > >>>>>> Is a managed (switch) involved? > >>>>> No, it is not. I do not use any switches in my testing setup. > >>>>> GW1--ISP1_modem--.....--ISP2_modem--GW2 > >>>>> > >>>>> Has duplex ever been an issue? > >>>>> I have never noticed any duplex issue. > >>>>> > >>>>> > >>>>> On Sun, 18 Aug 2019 16:07:14 -0500 > >>>>> Patrick Dohman <dohmanpatr...@gmail.com> wrote: > >>>>> > >>>>>> Does your ISP implement authoritative DNS? > >>>>>> Do you suspect a UDP issue? > >>>>>> Is a managed (switch) involved? Has duplex ever been an issue? > >>>>>> Regards > >>>>>> Patrick > >>>>>> > >>>>>>> On Aug 18, 2019, at 1:03 PM, Radek <r...@int.pl> wrote: > >>>>>>> > >>>>>>> Hello, > >>>>>>> > >>>>>>> I have two testing gateways (6.5/i386) with site-to-side VPN between > >>>>>>> its LANs (OpenIKED). > >>>>>>> Both gws are fully syspatched, have public IPs and the same iked/pf > >>>>>>> configuration. > >>>>>>> > >>>>>>> Unfortunately, the network traffic over the VPN tunnel stalls few > >>>>>>> times a day. > >>>>>>> > >>>>>>> On the one side I use a script to monitor VPN tunnel with ping, it > >>>>>>> restarts iked and emails me if there is no ping over the VPN tunnel. > >>>>>>> Date: Sat, 17 Aug 2019 22:10:30 +0200 (CEST) > >>>>>>> Date: Sun, 18 Aug 2019 06:00:20 +0200 (CEST) > >>>>>>> Date: Sun, 18 Aug 2019 11:09:00 +0200 (CEST) > >>>>>>> Date: Sun, 18 Aug 2019 19:03:02 +0200 (CEST) > >>>>>>> > >>>>>>> > >>>>>>> In 6.3/i386 I have the same problem, but more frequently. > >>>>>>> Date: Sat, 17 Aug 2019 23:03:56 +0200 (CEST) > >>>>>>> Date: Sun, 18 Aug 2019 01:37:50 +0200 (CEST) > >>>>>>> Date: Sun, 18 Aug 2019 04:12:31 +0200 (CEST) > >>>>>>> Date: Sun, 18 Aug 2019 06:46:25 +0200 (CEST) > >>>>>>> Date: Sun, 18 Aug 2019 09:20:22 +0200 (CEST) > >>>>>>> Date: Sun, 18 Aug 2019 11:59:08 +0200 (CEST) > >>>>>>> Date: Sun, 18 Aug 2019 14:34:38 +0200 (CEST) > >>>>>>> Date: Sun, 18 Aug 2019 17:12:57 +0200 (CEST) > >>>>>>> Date: Sun, 18 Aug 2019 19:47:16 +0200 (CEST) > >>>>>>> > >>>>>>> Do I have any bugs/deficiencies in my configs, missed something? > >>>>>>> Is there any way to make it work uninterruptedly? > >>>>>>> I would be very greatful if you could help me with this case. > >>>>>>> > >>>>>>> $cat /etc/hostname.enc0 > >>>>>>> up > >>>>>>> > >>>>>>> $cat /etc/hostname.vr3 > >>>>>>> inet 10.0.17.254 255.255.255.0 NONE description "LAN17" > >>>>>>> group trust > >>>>>>> > >>>>>>> $cat /etc/iked.conf > >>>>>>> local_gw_RAC17 = "10.0.17.254" # lan_RAC > >>>>>>> local_lan_RAC17 = "10.0.17.0/24" > >>>>>>> remote_gw_MON = "1.2.3.5" # fw_MON > >>>>>>> remote_lan_MON = "172.16.1.0/24" > >>>>>>> ikev2 quick active esp \ > >>>>>>> from $local_gw_RAC17 to $remote_gw_MON \ > >>>>>>> from $local_lan_RAC17 to $remote_lan_MON peer $remote_gw_MON \ > >>>>>>> childsa enc chacha20-poly1305 \ > >>>>>>> psk "psk" > >>>>>>> > >>>>>>> $cat /etc/pf.conf > >>>>>>> # RAC-fwTEST > >>>>>>> ext_if = "vr0" > >>>>>>> lan_rac_if = "vr3" # vr3 - > >>>>>>> lan_rac_local = $lan_rac_if:network # 10.0.17.0/24 > >>>>>>> backup_if = "vr2" # vr2 - lewy port > >>>>>>> backup_local = $backup_if:network # 10.0.117/24 > >>>>>>> > >>>>>>> bud = "1.2.3.0/25" > >>>>>>> rdk_wy = "1.2.3.4" > >>>>>>> rdk_mon = "1.2.3.5" > >>>>>>> panac_krz = "1.2.3.6" > >>>>>>> panac_rac = "1.2.3.7" > >>>>>>> > >>>>>>> set fingerprints "/dev/null" > >>>>>>> set skip on { lo, enc0 } > >>>>>>> set block-policy drop > >>>>>>> set optimization normal > >>>>>>> set ruleset-optimization basic > >>>>>>> > >>>>>>> antispoof quick for {lo0, $lan_rac_if, $backup_if } > >>>>>>> > >>>>>>> match out log on $ext_if from { $lan_rac_local, $backup_local } > >>>>>>> nat-to $ext_if set prio (3, 7) > >>>>>>> > >>>>>>> block all > >>>>>>> > >>>>>>> match in all scrub (no-df random-id) > >>>>>>> match out all scrub (no-df random-id) > >>>>>>> pass out on egress keep state > >>>>>>> > >>>>>>> pass from { 10.0.201.0/24, $lan_rac_local, $backup_local } to any set > >>>>>>> prio (3, 7) keep state > >>>>>>> > >>>>>>> ssh_port = "1071" > >>>>>>> table <ssh_trust> const { $bud, $rdk_wy, $rdk_mon, $panac_krz, > >>>>>>> $panac_rac, 10.0.2.0/24, 10.0.15.0/24, 10.0.100.0/24 } > >>>>>>> table <bruteforce> persist counters > >>>>>>> block from <bruteforce> > >>>>>>> pass in log quick inet proto tcp from <ssh_trust> to $ext_if port > >>>>>>> $ssh_port flags S/SA \ > >>>>>>> set prio (7, 7) keep state \ > >>>>>>> (max-src-conn 15, max-src-conn-rate 2/10, overload <bruteforce> > >>>>>>> flush global) > >>>>>>> > >>>>>>> icmp_types = "{ echoreq, unreach }" > >>>>>>> pass inet proto icmp all icmp-type $icmp_types \ > >>>>>>> set prio (7, 7) keep state > >>>>>>> > >>>>>>> table <vpn_peers> const { $rdk_mon, $panac_rac, $panac_krz } > >>>>>>> pass out quick on egress proto esp from (egress:0) to <vpn_peers> > >>>>>>> set prio (6, 7) keep state > >>>>>>> pass out quick on egress proto udp from (egress:0) to <vpn_peers> > >>>>>>> port {500, 4500} set prio (6, 7) keep state > >>>>>>> pass in quick on egress proto esp from <vpn_peers> to (egress:0) > >>>>>>> set prio (6, 7) keep state > >>>>>>> pass in quick on egress proto udp from <vpn_peers> to (egress:0) > >>>>>>> port {500, 4500} set prio (6, 7) keep state > >>>>>>> pass out quick on trust received-on enc0 set prio (6, 7) keep state > >>>>>>> > >>>>>>> pass in on egress proto udp from any to (egress:0) port > >>>>>>> {isakmp,ipsec-nat-t} set prio (6,7) keep state > >>>>>>> pass in on egress proto {ah,esp} set prio (6,7) keep state > >>>>>>> > >>>>>>> # By default, do not permit remote connections to X11 > >>>>>>> block return in on ! lo0 proto tcp to port 6000:6010 > >>>>>>> > >>>>>>> $cat iked_monitor.sh > >>>>>>> #!/bin/sh > >>>>>>> while true > >>>>>>> do > >>>>>>> vpn=`ping -c 3 -w 1 -I 10.0.17.254 172.16.1.254 | grep packets | awk > >>>>>>> -F " " '{print $4}'` > >>>>>>> > >>>>>>> if [ "${vpn}" -eq 0 ] ; then > >>>>>>> mon=`ping -c 3 -w 1 the_other_side_WAN_IP | grep packets | awk -F " " > >>>>>>> '{print $4}'` > >>>>>>> wan=`ping -c 3 -w 1 8.8.8.8 | grep packets | awk -F " " '{print $4}'` > >>>>>>> > >>>>>>> if [ "${mon}" -gt 0 ] && [ "${wan}" -gt 0 ] ; then > >>>>>>> echo vpn: ${vpn}, mon: ${mon}, wan: ${wan} | mail -s "no ping > >>>>>>> through VPN RACTEST-MON! restartng iked!" em...@example.com > >>>>>>> rcctl restart iked > >>>>>>> fi > >>>>>>> fi > >>>>>>> sleep 32 > >>>>>>> done > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Radek > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Radek > >>>>> > >>>> > >>> > >>> > >>> -- > >>> Radek > >>> > >> > > > > > > -- > > Radek > -- Radek