Here is how I reproduce the bug:
Create 3 Ubuntu 16.04 VMs (VM-1, VM gateway, VM-2) on Azure in the same
Resource Group. The kernel should be the linux-azure kernel
4.15.0-1098.109~16.04.1 (or newer). I use Gen1 VM but Gen2 should also
has the same issue; I use the "East US 2" region, but the issue should
reproduce in any region.
Note: none of the VMs use Accelerated Networking, i.e. all of the 3 VMs
use the software NIC "NetVSC".
In my setup, VM-1 and VM-2 are "Standard D8s v3 (8 vcpus, 32 GiB
memory)", and VM-gateway is "Standard DS3 v2 (4 vcpus, 14 GiB memory)".
I happen to name the gateway VM "decui-dpdk", but here actually DPDK is
not used at all (I do intend to use this setup for DPDK in future).
The gateway VM has 3 NICs:
The main NIC (10.0.0.4) is not used in ip-forwarding.
NIC-1's IP is 192.168.80.5
NIC-2's IP is 192.168.81.5.
The gateway VM receives packets from VM-1(192.168.80.4) and forwards
the packets to VM-2 (192.168.81.4).
No firewall rule is used.
The client VM (VM-1, 192.168.80.4) has 1 NIC. It's running iperf2 client.
The server VM (VM-2, 192.168.81.4) has 1 NIC. It's running iperf2 server:
"nohup iperf -s &"
The client VM is sending traffic, through the gateway VM (192.168.80.5,
192.168.81.5), to the server VM.
Note: all the 3 subnets here are in the same VNET(Virtual Net) and 2 Azure UDR
(User Defined Routing) rules must be used to force the traffic to go through
the gateway VM. The IP-forwarding of the gateway VM's NIC-1 and NIC-2 must be
enabled from Azure portal (the setting can only changed when the VM is
"stopped"), and IP-forwarding must be enabled in the gateway VM (i.e. echo 1 >
/proc/sys/net/ipv4/ip_forward). I'll attach some screenshots showing the
network topology and the configuration.
iperf2 uses 512 TCP connections and I limit the bandwidth used by iperf to
<=70% of the per-VM bandwith limit (Note: if the VM uses >70% of the limit,
even with the 2 patches, the ping latency between VM-1 and VM-2 can still
easily go very high, e.g. >200ms -- we'll try to further investigate that).
It looks the per-VM bandwithd limit of the gateway VM (DS3_v2) is 2.6Gbps, so
70% of it is 1.8Gbps.
In the client VM, run something like:
iperf -c 192.168.81.4 -b 3.5m -t 120 -P512
(-b means the per-TCP-connection limit; -P512 means 512 connections, so the
total throughput should be around 3.5*512 = 1792 Mbps; "-t 120" means the test
lasts for 2 minutes. we can abort the test any time by Ctrl+C.)
In the "Server VM, run:
nohup iperf -s &
ping 192.168.80.4 (we can terminate the program by Ctrl+C), and observe the
latency.
In the gateway VM, run "nload" to check the current throughput (if the
current device is not the NIC we want to check, press Right Arrow and
Left Allow), and run "top" to check the CPU utilization (when there are
512 connections, the utilization should be still low, e.g. <25%).
When the iperf2 test is running, the ping latency between VM-1 and VM-2
can easily exceed 100ms or even >300ms, but with the 2 patches applied,
the latency typically should be <20ms.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1902531
Title:
[linux-azure] IP forwarding issue in netvsc
Status in linux-azure package in Ubuntu:
New
Bug description:
We identified an issue with the Linux netvsc driver when used in IP
forwarding mode. The problem is that the RSS hash value is not
propagated to the outgoing packet, and so such packets go out on
channel 0. This produces an imbalance across outgoing channels, and a
possible overload on the single host-side CPU that is processing
channel 0. The problem does not occur when Accelerated Networking is
used because the packets go out through the Mellanox driver. Because
it is tied to IP forwarding, the problem is presumably most likely to
be visible in a virtual appliance device that is doing network load
balancing or other kinds of packet filtering and redirection.
We would like to request fixes to this issue in 16.04, 18.04 and
20.04.
Two fixes are already in the upstream v5.5+, so they’re already in
5.8.0-1011.11.
For 5.4.0-1031.32, the 2 fixes can apply cleanly:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1fac7ca4e63bf935780cc632ccb6ba8de5f22321
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6f3aeb1ba05d41320e6cf9a60f698d9c4e44348e
For 5.0.0-1036.38, we need 1 more patch applied first, so the list is:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b441f79532ec13dc82d05c55badc4da1f62a6141
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1fac7ca4e63bf935780cc632ccb6ba8de5f22321
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6f3aeb1ba05d41320e6cf9a60f698d9c4e44348e
For 4.15.0-1098.109~16.04.1, the 2 patches can not apply cleanly, so Dexuan
backported them here:
https://github.com/dcui/linux/commit/4ed58762a56cccfd006e633fac63311176508795
https://github.com/dcui/linux/commit/40ad7849a6365a5a485f05453e10e3541025e25a
(The 2 patches are on the branch
https://github.com/dcui/linux/commits/decui/ubuntu_16.04/linux-azure/Ubuntu-azure-4.15.0-1098.109_16.04.1)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1902531/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
