You went from emulated Realtek ethernet to xnf. Can you try other network interfaces?
Berry Wendermouth [bayb...@riseup.net] wrote: > Xen based VPS / OpenBSD 6.2 / OpenVPN 2.4.4 => Slow download speed after > upgrade > ================================================================================ > > Dear OpenBSD Community, > > we are operating an OpenVPN server on OpenBSD. A few days ago we > upgraded to OpenBSD 6.2 > and we are now seeing very slow speeds (<10KB/s) when trying to download > via > the VPN tunnel from the internet (WAN). We did not have this problem > before. > > >From the documented test cases below (Specifically case 2) it does not > look like it is a VPN performance problem (e.g. mtu/encryption > performance related). > We can also exclude bandwidth trottleing by the VPS provider and the > ISP. > > * Did something essential change in `pf`? [4] > * Or is the problem related to OpenBSD's Xen drivers? > > Could someone help us track down the bottleneck? > > Any help and hints are very much appreciated. > > Thank you kindly > > Berry > > PS: for a better viewing experience you may compile this email body with > `asciidoc` > > == Environment > > === Server > * OpenBSD 6.2 / amd64 (-release) + syspatch > * OpenVPN 2.4.4 > * On Virtual Private Server / Xen version "4.9.0" by Xen Project [0] > * Detected CPU: Intel(R) Xeon(R) CPU E5-2620 > * Detected network device: xnf0 > * Firewall configuration: /etc/pf.conf [1] > * System Message Buffer [2] > > === Clients > * OpenBSD 6.2 with OpenVPN 2.4.4 > * GNU/Linux Gentoo with OpenVPN 2.4.4 > * LinesageOS 14.1 with OpenVPN for Android 0.6.73 > > == Detailed Problem Description / Test Results > > Please note: the following documented tests used one and the same client > / network connection: > > * GNU/Linux Gentoo with OpenVPN 2.4.4 > * Connected to router via wifi on internet connection with max 50Mbit/s > download > > To rule out problems with the client local network settings tests with > other client setups on other networks were also performed and showed > identical > results. For brevity they are not documented here. > > === Case 1: Server <==> WAN (ok) > * When on the server, downloading a file from WAN > * Scenario: downloaded 100MB file from > http://fra36-speedtest-1.tele2.net/ with curl > * Average Download Speed: ~ 10Mbit/s > * Testresult: > > ---- > $ curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left Speed > 100 100M 100 100M 0 0 9309k 0 0:00:11 0:00:11 --:--:-- > 10.9M > ---- > > === Case 2: Client <= VPN => Server (ok) > * When on the client, downloading a file from server via VPN tunnel > * Scenario: standard download test with `iperf` > * Average Download Speed: ~ 15Mbit/s > * Testresult: > > ---- > # iperf -s > > > --- > Server listening on TCP port 5001 > TCP window size: 16.0 KByte (default) > --- > [ 4] local 10.8.0.1 port 5001 connected with 10.8.0.4 port 34998 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.2 sec 18.5 MBytes 15.2 Mbits/sec > > > # iperf -c 10.8.0.1 > --- > Client connecting to 10.8.0.1, TCP port 5001 > TCP window size: 45.0 KByte (default) > --- > [ 3] local 10.8.0.4 port 34998 connected with 10.8.0.1 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 18.5 MBytes 15.5 Mbits/sec > ---- > > === Case 3a: Client <= VPN => Server <==> WAN (broken) > * When on the client, downloading a file from WAN via VPN tunnel > * Scenario: downloaded 100MB file from > http://fra36-speedtest-1.tele2.net/ with curl > * Average Download Speed: ~ 5KB/s > * Testresult: > > ---- > curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left Speed > 0 100M 0 149k 0 0 5102 0 5:42:32 0:00:30 5:42:02 > 4933 > ---- > > === Case 3b: Client <==> WAN (ok) > * When on the client, downloading a file from WAN directly > * Scenario: downloaded 100MB file from > http://fra36-speedtest-1.tele2.net/ with curl > * Average Download Speed: ~ 1100KB/s > * Testresult: > > ---- > curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left Speed > 100 100M 100 100M 0 0 1113k 0 0:01:32 0:01:32 --:--:-- > 1196k > ---- > > == Previous working system > Before the upgrade to OpenBSD 6.2 we had a working system with the > following setup: > > * OpenBSD 6.1 / i386 > * OpenVPN 2.4.1 > * firewall settings were the same [8] > > The fact that we had installed i386 instead of amd64 was unintentional. > > We had to change the virtual machine (QEMU) network interface from > Realtek to > Virtio to get a good performance on the external network interface. > Hence > the working system's external interface was operating on `vio`. The > following > system message buffer still lists the inefficient `re` device. > > * System Message Buffer [3] > > == Appendix > * [0] https://www.xenproject.org/ > * [1] Firewall configuration: /etc/pf.conf > ---- > ext_if="xnf0" > vpn_if="tun0" > vpn_ip="10.8.0.1" > vpn_sn="10.8.0.0/24" > server="10.8.0.99" > > ssh_port="22" > vpn_port="1094" > iperf_port="5001" > server_tcp_ip4_ports="{ 25, 53, 80, 443, 465, 587, 993, 5222, 5269, 9999 > }" > server_udp_ip4_ports="{ 53, 5353, 67 }" > > # Runtime Options > set block-policy return > set loginterface egress > set skip on lo > > #block log all > match in all scrub (no-df max-mss 1440 random-id) > > # forwarding from WAN through tunnel to client > pass in quick on $ext_if proto { tcp } from any to ($ext_if) port > $server_tcp_ip4_ports rdr-to $server > pass in quick on $ext_if proto { udp } from any to ($ext_if) port > $server_udp_ip4_ports rdr-to $server > > # route outwards from tunnel > pass out quick on $ext_if from $vpn_sn to any nat-to ($ext_if) > > # incoming > pass in quick on $ext_if proto { tcp } from any to ($ext_if) port { > $ssh_port $iperf_port } flags S/SA synproxy state > pass in quick on $ext_if proto { udp } from any to ($ext_if) port { > $ssh_port $vpn_port $iperf_port } > block drop in quick on $ext_if all > > # out to WAN > pass out quick on $ext_if from ($ext_if) to any modulate state > block drop out quick on $ext_if all > ---- > > * [2] system message buffer 6.2: > ---- > openBSD 6.2 (GENERIC) #0: Thu Oct 12 19:16:36 CEST 2017 > r...@syspatch-62-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC > real mem = 2122313728 (2023MB) > avail mem = 2051125248 (1956MB) > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfc001000 (11 entries) > bios0: vendor Xen version "4.9.0" date 09/10/2017 > bios0: Xen HVM domU > acpi0 at bios0: rev 2 > acpi0: sleep states S3 S4 S5 > acpi0: tables DSDT FACP APIC HPET WAET SSDT SSDT > acpi0: wakeup devices > acpitimer0 at acpi0: 3579545 Hz, 32 bits > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 11, 48 pins > , remapped to apid 1 > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz, 2100.27 MHz > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,FSGSBASE,SMEP,ERMS > cpu0: 256KB 64b/line 8-way L2 cache > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges > cpu0: apic clock running at 99MHz > acpihpet0 at acpi0: 62500000 Hz > acpiprt0 at acpi0: bus 0 (PCI0) > acpicpu0 at acpi0: C1(@1 halt!) > "PNP0F13" at acpi0 not configured > "PNP0700" at acpi0 not configured > "ACPI0007" at acpi0 not configured > pvbus0 at mainbus0: Xen 4.9 > xen0 at pvbus0: features 0x2705, 32 grant table frames, event channel 1 > xbf0 at xen0 backend 0 channel 5: disk > scsibus1 at xbf0: 2 targets > sd0 at scsibus1 targ 0 lun 0: <Xen, phy xvda 51712, 0000> SCSI3 0/direct > fixed > sd0: 51200MB, 512 bytes/sector, 104857600 sectors > xbf1 at xen0 backend 0 channel 6: cdrom > scsibus2 at xbf1: 2 targets > cd0 at scsibus2 targ 0 lun 0: <Xen, qdisk xvdc 5174, 0000> SCSI3 5/cdrom > fixed > "vkbd" at xen0: device/vkbd/0 not configured > xnf0 at xen0 backend 0 channel 7: address 00:50:56:34:10:49 > pci0 at mainbus0 bus 0 > pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02 > pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00 > pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, > channel 0 wired to compatibility, channel 1 wired to compatibility > pciide0: channel 0 disabled (no drives) > atapiscsi0 at pciide0 channel 1 drive 0 > scsibus3 at atapiscsi0: 2 targets > cd1 at scsibus3 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> ATAPI 5/cdrom > removable > cd1(pciide0:1:0): using PIO mode 4, DMA mode 2 > uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 1 int > 23 > piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: SMBus > disabled > xspd0 at pci0 dev 2 function 0 "XenSource Platform Device" rev 0x01 > vga1 at pci0 dev 3 function 0 "Cirrus Logic CL-GD5446" rev 0x00 > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) > wsdisplay0: screen 1-5 added (80x25, vt100 emulation) > isa0 at pcib0 > isadma0 at isa0 > fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > pckbc0 at isa0 port 0x60/5 irq 1 irq 12 > pckbd0 at pckbc0 (kbd slot) > wskbd0 at pckbd0: console keyboard, using wsdisplay0 > pms0 at pckbc0 (aux slot) > wsmouse0 at pms0 mux 0 > pcppi0 at isa0 port 0x61 > spkr0 at pcppi0 > usb0 at uhci0: USB revision 1.0 > uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev > 1.00/1.00 addr 1 > uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB > Tablet" rev 2.00/0.00 addr 2 > uhidev0: iclass 3/0 > ums0 at uhidev0: 3 buttons, Z dir > wsmouse1 at ums0 mux 0 > vscsi0 at root > scsibus4 at vscsi0: 256 targets > softraid0 at root > scsibus5 at softraid0: 256 targets > root on sd0a (244889b124e5edd0.a) swap on sd0b dump on sd0b > fd0 at fdc0 drive 1: density unknown > ---- > > * [3] Working system message buffer before upgrade from 6.1 to 6.2 > ---- > OpenBSD 6.1 (GENERIC) #291: Sat Apr 1 13:49:08 MDT 2017 > dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC > cpu0: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz ("GenuineIntel" > 686-class) 2.11 GHz > cpu0: > FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,NXE,PAGE1GB,LONG,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,LAHF,FSGSBASE,SMEP,ERMS > real mem = 2138583040 (2039MB) > avail mem = 2084909056 (1988MB) > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: date 06/23/99, BIOS32 rev. 0 @ 0xfd578, SMBIOS rev. > 2.4 @ 0xfc001000 (11 entries) > bios0: vendor Xen version "4.9.0" date 09/10/2017 > bios0: Xen HVM domU > acpi0 at bios0: rev 2 > acpi0: sleep states S3 S4 S5 > acpi0: tables DSDT FACP APIC HPET WAET SSDT SSDT > acpi0: wakeup devices > acpitimer0 at acpi0: 3579545 Hz, 32 bits > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 11, 48 pins > cpu0 at mainbus0: apid 0 (boot processor) > mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges > cpu0: apic clock running at 100MHz > acpihpet0 at acpi0: 62500000 Hz > acpiprt0 at acpi0: bus 0 (PCI0) > acpicpu0 at acpi0: C1(@1 halt!) > "PNP0F13" at acpi0 not configured > "PNP0303" at acpi0 not configured > "PNP0700" at acpi0 not configured > "PNP0501" at acpi0 not configured > "ACPI0007" at acpi0 not configured > bios0: ROM list: 0xc0000/0x9600 0xc9800/0xe00 0xec000/0x4000! > pvbus0 at mainbus0: Xen 4.9 > pci0 at mainbus0 bus 0: configuration mode 1 (bios) > pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02 > pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00 > pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, > channel 0 wired to compatibility, channel 1 wired to compatibility > wd0 at pciide0 channel 0 drive 0: <QEMU HARDDISK> > wd0: 16-sector PIO, LBA48, 51200MB, 104857600 sectors > wd0(pciide0:0:0): using PIO mode 4, DMA mode 2 > atapiscsi0 at pciide0 channel 1 drive 0 > scsibus1 at atapiscsi0: 2 targets > cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> ATAPI 5/cdrom > removable > cd0(pciide0:1:0): using PIO mode 4, DMA mode 2 > uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 1 int > 23 > piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: SMBus > disabled > "XenSource Platform Device" rev 0x01 at pci0 dev 2 function 0 not > configured > vga1 at pci0 dev 3 function 0 "Cirrus Logic CL-GD5446" rev 0x00 > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) > wsdisplay0: screen 1-5 added (80x25, vt100 emulation) > re0 at pci0 dev 4 function 0 "Realtek 8139" rev 0x20: RTL8139C+ > (0x7480), apic 1 int 32, address 00:50:56:34:10:49 > rlphy0 at re0 phy 0: RTL internal PHY > isa0 at pcib0 > isadma0 at isa0 > fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 > fd0 at fdc0 drive 1: density unknown > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > pckbc0 at isa0 port 0x60/5 irq 1 irq 12 > pckbd0 at pckbc0 (kbd slot) > wskbd0 at pckbd0: console keyboard, using wsdisplay0 > pms0 at pckbc0 (aux slot) > wsmouse0 at pms0 mux 0 > pcppi0 at isa0 port 0x61 > spkr0 at pcppi0 > npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 > usb0 at uhci0: USB revision 1.0 > uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev > 1.00/1.00 addr 1 > nvram: invalid checksum > uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB > Tablet" rev 2.00/0.00 addr 2 > uhidev0: iclass 3/0 > ums0 at uhidev0: 3 buttons, Z dir > wsmouse1 at ums0 mux 0 > vscsi0 at root > scsibus2 at vscsi0: 256 targets > softraid0 at root > scsibus3 at softraid0: 256 targets > root on wd0a (244889b124e5edd0.a) swap on wd0b dump on wd0b > clock: unknown CMOS layout > ---- > > * [4] https://www.openbsd.org/62.html - search for "Generic network > stack improvements"
