Hi all, I'm currently working on a Qubes OS server version (example architecture can been seen at https://raw.githubusercontent.com/fepitre/qubes-mgmt-salt-qubes-server/devel-140320-extra/qubes-server.png). I'm using this configuration since several months on Qubes R4.0 (xen-4.8) and recently on Qubes R4.1 (xen-4.13). I'm writing to you because since the beginning I'm having network performance issues that I never succeeded to solve.
This setup is done on a HP Gen8 DL360p with 2*CPUs, 160GB memory, 1TB RAID6 SAS.
On the picture I linked you, all the colored rectangles {zone}-* for zone in
(wan, dmz, lan, admin) are PVH VMs (Debian 10). There exist a VM not drawn in
the picture, called 'sys-net-interfaces' which holds four 1Gbits Ethernet
controllers of the server using PCI passthrough. It is a HVM with Linux-based
stubdomain.
All the inner links between VMs are NAT interfaces. All the outer links on
*-sys-net VMs are BRIDGE interfaces with backend 'sys-net-interfaces'. In VM
'sys-net-interfaces' a LACP bond0 is done with two Ethernet controllers, which
is a trunk, then several vlan interfaces are generated with parent device this
bond, and finally, bridges are created and associated to those vlans.
Here are my issues. Consider one computer named 'PC-LAN' in LAN network and
another 'PC-DMZ' in DMZ network. The considered network path is the following:
PC-LAN (1) <-- B --> lan-sys-net (2) <-- N --> lan-sys-firewall (3) <--
N --> dmz-sys-firewall (4) <-- N --> dmz-sys-net (5) <-- B --> PC-DMZ (6)
where B denotes bridge interface, N denotes NAT interface and numbers for
numbering machines. Up to 'wget', 'scp' (limited normally by ciphers), etc., I
ran multiple iperf3 tests over 20 seconds for having a clearer view of network
issues.
Example 1: Full path
From (1) to (6): 165 Mbits/s
From (2) to (6): 196 Mbits/s
From (3) to (6): 205 Mbits/s
From (4) to (6): 203 Mbits/s
From (5) to (6): 714 Mbits/s
Example 2: 'dmz-sys-net' as end node
From (1) to (5): 194 Mbits/s
From (2) to (5): 189 Mbits/s
From (3) to (5): 258 Mbits/s
From (4) to (5): 500 Mbits/s
Example 3: 'lan-sys-net' as end node
From (1) to (2): 830 Mbits/s
I've another HP Gen8 with almost the same physical configuration and network
configuration (LACP+vlan+bridges) running under Debian 10 as bare metal KVM,
and I obtain 1Gbits/s network workflows over bridges. The almost physical
configuration is due to the related mail I sent you in july 2019 '[Xen-devel]
Ethernet PCI passthrough problem'. The provided Ethernet card with 4 ports (HP
Ethernet 1Gb 4-port 331FLR Adapter) makes the driver tg3 crashing when
attaching those into a VM. So the Debian KVM has those HP Ethernet controllers
whereas on the Qubes server, it has a cheap PCI express 4 Ethernet Realtek 8169
card.
Of course physical connections on the switches have been changed, 'switched'
between the two servers for eliminating any hardware problem.
I had a look to
https://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide.
Unfortunately, trying some change of options with 'ethtool' in
'sys-net-interfaces', changing amount of RAM/VCPUs of it and other *-sys-net,
does not do that much.
I'm writing to you for having some clues into where I can dig and what I can
look in order to put in evidence the bottleneck. If it's purely dom0 side or
backend network VM side (sys-net-interfaces) or elsewhere.
I would like to thank you a lot in advance for any help on this problem.
Best regards,
Frédéric
signature.asc
Description: OpenPGP digital signature
