** Description changed: - I've ran into an issue when creating a large number of virtual functions - on a SR-IOV capable device. + [Impact] - ip link show reports a message truncated error: + When querying a Physical Function netdev with a large amount of VF's + (more than 30), the resulting return message can overflow the 16K + netlink message buffer. + + This can be fixed by enabling message peeking on the socket and resizing + the buffer on receive, or by simply enlarging the receive buffer. + + Since there's an upper limit to the number of VF's per PF, it's + relatively sane to just enlarge the receive buffer. Please see the + attached patch. + + [Test Case] + + # Set up 60 VF's on an SR-IOV device ip link show > /dev/null Message truncated Message truncated Message truncated - A likely cause might be that when called in a system where the number of - PCIe Virtual Functions are more than 30 for a given Physical Function, - the netlink response is larger than 16K, meaning that a message is - truncated. + [Regression Potential] - The issue is seen with Ubuntu14.04 and Ubuntu16.04. + 1) Applications relying on the broken behaviour will need to be updated, but it would be a really dubious use case. + 2) Increasing the rx buffer size increases the memory footprint (but realistically, this is tiny). + 3) Extra processing time is now needed to parse the larger buffer, in the case that a call to "ip link" is on the critical time path of an application, (called multiple times in a tight loop, for example), it would affect load. - A possible solution for the issue is to increase the size of the receive - buffer in libnetlink.c + All of the above are mitigated because this workaround is in upstream + iproute2 and has been running on different distributions without + repercussions. - Additional information: + [Other Info] + + Observed on Ubuntu kernel 4.4.0-93-generic on both 14.04 and 16.04 + ===================================================================================================== Ubuntu16 system stack@cluster04:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial stack@cluster04:~$ uname -r 4.4.0-93-generic stack@cluster04:~$ apt-cache policy iproute2 iproute2: - Installed: 4.3.0-1ubuntu3.16.04.1 + Installed: 4.3.0-1ubuntu3.16.04.1 Version table: *** 4.3.0-1ubuntu3.16.04.1 500 - 500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages + 500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages ================================================================================================= Ubuntu14 system: root@boomslang:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty root@boomslang:~# uname -r 4.4.0-96-generic root@boomslang:~# apt-cache policy iproute2 iproute2: - Installed: 3.12.0-2ubuntu1 - Version table: - *** 3.12.0-2ubuntu1 0 - 500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages + Installed: 3.12.0-2ubuntu1 + Version table: + *** 3.12.0-2ubuntu1 0 + 500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
** Description changed: [Impact] When querying a Physical Function netdev with a large amount of VF's (more than 30), the resulting return message can overflow the 16K netlink message buffer. This can be fixed by enabling message peeking on the socket and resizing the buffer on receive, or by simply enlarging the receive buffer. Since there's an upper limit to the number of VF's per PF, it's relatively sane to just enlarge the receive buffer. Please see the attached patch. [Test Case] # Set up 60 VF's on an SR-IOV device ip link show > /dev/null + + Observe the following: Message truncated Message truncated Message truncated [Regression Potential] 1) Applications relying on the broken behaviour will need to be updated, but it would be a really dubious use case. 2) Increasing the rx buffer size increases the memory footprint (but realistically, this is tiny). 3) Extra processing time is now needed to parse the larger buffer, in the case that a call to "ip link" is on the critical time path of an application, (called multiple times in a tight loop, for example), it would affect load. All of the above are mitigated because this workaround is in upstream iproute2 and has been running on different distributions without repercussions. [Other Info] Observed on Ubuntu kernel 4.4.0-93-generic on both 14.04 and 16.04 ===================================================================================================== Ubuntu16 system stack@cluster04:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial stack@cluster04:~$ uname -r 4.4.0-93-generic stack@cluster04:~$ apt-cache policy iproute2 iproute2: Installed: 4.3.0-1ubuntu3.16.04.1 Version table: *** 4.3.0-1ubuntu3.16.04.1 500 500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages ================================================================================================= Ubuntu14 system: root@boomslang:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty root@boomslang:~# uname -r 4.4.0-96-generic root@boomslang:~# apt-cache policy iproute2 iproute2: Installed: 3.12.0-2ubuntu1 Version table: *** 3.12.0-2ubuntu1 0 500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages ** Description changed: [Impact] When querying a Physical Function netdev with a large amount of VF's (more than 30), the resulting return message can overflow the 16K netlink message buffer. This can be fixed by enabling message peeking on the socket and resizing the buffer on receive, or by simply enlarging the receive buffer. Since there's an upper limit to the number of VF's per PF, it's relatively sane to just enlarge the receive buffer. Please see the attached patch. [Test Case] # Set up 60 VF's on an SR-IOV device ip link show > /dev/null Observe the following: Message truncated Message truncated Message truncated [Regression Potential] 1) Applications relying on the broken behaviour will need to be updated, but it would be a really dubious use case. 2) Increasing the rx buffer size increases the memory footprint (but realistically, this is tiny). 3) Extra processing time is now needed to parse the larger buffer, in the case that a call to "ip link" is on the critical time path of an application, (called multiple times in a tight loop, for example), it would affect load. - - All of the above are mitigated because this workaround is in upstream - iproute2 and has been running on different distributions without - repercussions. [Other Info] Observed on Ubuntu kernel 4.4.0-93-generic on both 14.04 and 16.04 ===================================================================================================== Ubuntu16 system stack@cluster04:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial stack@cluster04:~$ uname -r 4.4.0-93-generic stack@cluster04:~$ apt-cache policy iproute2 iproute2: Installed: 4.3.0-1ubuntu3.16.04.1 Version table: *** 4.3.0-1ubuntu3.16.04.1 500 500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages ================================================================================================= Ubuntu14 system: root@boomslang:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty root@boomslang:~# uname -r 4.4.0-96-generic root@boomslang:~# apt-cache policy iproute2 iproute2: Installed: 3.12.0-2ubuntu1 Version table: *** 3.12.0-2ubuntu1 0 500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720126 Title: [ip link] Message truncated error for large number of passthrough VFs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
