OpenBSD 6.2 amd64 on "Xen Project": Very slow download rate via VPN ===================================================================
Hello everybody. We are operating an OpenBSD instance serving OpenVPN. The system is run as a guest in a Xen Project virtual machine. When conncted from a VPN client we are getting very slow throughput when downloading data from the WAN (<= 10 KB/s). Disabling the default xnf driver in the OpenBSD kernel and using virtio instead improves the performance fundamentally. I'm reporting this bug in reference to recommendation in #openbsd on IRC. == Environment === Server * On Virtual Private Server / Xen version "4.9.0" by Xen Project * Guest Operating System: OpenBSD 6.2 / amd64 (-release) + syspatch * OpenVPN 2.4.4 * Firewall configuration [1] * System Message Buffer [2] === Clients * OpenBSD 6.2 with OpenVPN 2.4.4 * GNU/Linux Gentoo with OpenVPN 2.4.4 * LinesageOS 14.1 with OpenVPN for Android 0.6.73 == Use Case Description === Preconditions * Logged in on a client that is connected to VPN server * Client has sufficient connection to external network * Server has sufficient connection to external network * Download data source has sufficient connection to external network === Execution * Download file from Internet (WAN), for example: ---- curl http://fra36-speedtest-1.tele2.net/100MB.zip > /dev/null ---- === Our Test Results 1. With xnf driver enabled: download speed is <= 10 KB/s 2. With xnf driver disabled: download speed is ~ 1 MB/s Also see [3] for an extensive description of our test results including documented * download performance directly from server * download performance from client without VPN * download performance from client with VPN with xnf driver enabled * download performance from client with VPN with xnf driver disabled === Similar cases * With commercial implementation of XenServer 6.5 [4] == Additional Notes * When the xnf driver is disabled it falls back to the network model defined in virtual machine config, for example [5] == Appendix * [1] Firewall configuration: /etc/pf.conf ---- ext_if="xnf0" vpn_if="tun0" vpn_ip="10.8.0.1" vpn_sn="10.8.0.0/24" server="10.8.0.99" ssh_port="22" vpn_port="1094" iperf_port="5001" server_tcp_ip4_ports="{ 25, 53, 80, 443, 465, 587, 993, 5222, 5269, 9999 }" server_udp_ip4_ports="{ 53, 5353, 67 }" # Runtime Options set block-policy return set loginterface egress set skip on lo #block log all match in all scrub (no-df max-mss 1440 random-id) # forwarding from WAN through tunnel to client pass in quick on $ext_if proto { tcp } from any to ($ext_if) port $server_tcp_ip4_ports rdr-to $server pass in quick on $ext_if proto { udp } from any to ($ext_if) port $server_udp_ip4_ports rdr-to $server # route outwards from tunnel pass out quick on $ext_if from $vpn_sn to any nat-to ($ext_if) # incoming pass in quick on $ext_if proto { tcp } from any to ($ext_if) port { $ssh_port $iperf_port } flags S/SA synproxy state pass in quick on $ext_if proto { udp } from any to ($ext_if) port { $ssh_port $vpn_port $iperf_port } block drop in quick on $ext_if all # out to WAN pass out quick on $ext_if from ($ext_if) to any modulate state block drop out quick on $ext_if all ---- * [2] system message buffer 6.2: ---- openBSD 6.2 (GENERIC) #0: Thu Oct 12 19:16:36 CEST 2017 [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC real mem = 2122313728 (2023MB) avail mem = 2051125248 (1956MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfc001000 (11 entries) bios0: vendor Xen version "4.9.0" date 09/10/2017 bios0: Xen HVM domU acpi0 at bios0: rev 2 acpi0: sleep states S3 S4 S5 acpi0: tables DSDT FACP APIC HPET WAET SSDT SSDT acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 11, 48 pins , remapped to apid 1 cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz, 2100.27 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MM \ X,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,FSGSBASE,SMEP,ERMS cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz acpihpet0 at acpi0: 62500000 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0: C1(@1 halt!) "PNP0F13" at acpi0 not configured "PNP0700" at acpi0 not configured "ACPI0007" at acpi0 not configured pvbus0 at mainbus0: Xen 4.9 xen0 at pvbus0: features 0x2705, 32 grant table frames, event channel 1 xbf0 at xen0 backend 0 channel 5: disk scsibus1 at xbf0: 2 targets sd0 at scsibus1 targ 0 lun 0: <Xen, phy xvda 51712, 0000> SCSI3 0/direct fixed sd0: 51200MB, 512 bytes/sector, 104857600 sectors xbf1 at xen0 backend 0 channel 6: cdrom scsibus2 at xbf1: 2 targets cd0 at scsibus2 targ 0 lun 0: <Xen, qdisk xvdc 5174, 0000> SCSI3 5/cdrom fixed "vkbd" at xen0: device/vkbd/0 not configured xnf0 at xen0 backend 0 channel 7: address 00:50:56:34:10:49 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02 pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00 pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility pciide0: channel 0 disabled (no drives) atapiscsi0 at pciide0 channel 1 drive 0 scsibus3 at atapiscsi0: 2 targets cd1 at scsibus3 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> ATAPI 5/cdrom removable cd1(pciide0:1:0): using PIO mode 4, DMA mode 2 uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 1 int 23 piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: SMBus disabled xspd0 at pci0 dev 2 function 0 "XenSource Platform Device" rev 0x01 vga1 at pci0 dev 3 function 0 "Cirrus Logic CL-GD5446" rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) isa0 at pcib0 isadma0 at isa0 fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0: console keyboard, using wsdisplay0 pms0 at pckbc0 (aux slot) wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 usb0 at uhci0: USB revision 1.0 uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 2.00/0.00 addr 2 uhidev0: iclass 3/0 ums0 at uhidev0: 3 buttons, Z dir wsmouse1 at ums0 mux 0 vscsi0 at root scsibus4 at vscsi0: 256 targets softraid0 at root scsibus5 at softraid0: 256 targets root on sd0a (244889b124e5edd0.a) swap on sd0b dump on sd0b fd0 at fdc0 drive 1: density unknown ---- * [3] https://marc.info/?t=150938594600011&r=1&w=2 * [4] http://daemonforums.org/showthread.php?p=61158 * [5] "Xen Project" version 4.9.0 Virtual machine * config extract ---- vif = [ 'vifname=some-name, model=virtio-net, rate=100Mb/s, bridge=xenbr0.781, mac=00:x:x:x:x:x, ip=x.x.x.x x:x:0:0:0:0:3:7' ] ----
