** Package changed: dkms (Ubuntu) => linux (Ubuntu)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1621474
Title:
VLAN creation fails with VF multi queueuing
Status in linux package in Ubuntu:
New
Bug description:
Description
------------
It can be read in SR-IOV companion guide that 82599 chipsets :
* The 82599 has 128 Transmit and 128 Receive queues. They are generally
referred to/thought of as queue pairs (1 Transmit and 1 Receive queue). This
gives the 82599 128 queue pairs.
* The 82599 has up to 64 pools, with each pool having two queue pairs in
it. That is two Transmit and Receive queue assigned to each VF.
* The 82599 allows for a variable configuration for the number of pools.
In SR-IOV mode, there can be 16, 32 or 64 pools.
The last line suggest using combined queues of 2, 4 and 8.
The only way, for now we found, is using tc on the host especially as the
ethtool command is not supported (ethtool -L eth10 combined 8 for example).
And, with following command "tc qdisc add dev eth10 root mqprio num_tc 8", we
will be able to use up to 8 combined queues on the host PF and guests which are
using VFs.
Yet, with 8 combined queues with tc commnad, VLAN creation was not permitted.
Following driver versions have been used on host and guest to enable this
multiqueue option:
* 4.3.15 ixgbe in the host
* 3.1.2 ixgbevf in the guest
To reproduce
-------------
On the host side
----------------
root@ubuntu1404:~# rmmod ixgbe
root@ubuntu1404:~# modprobe ixgbe
[ 4817.027637] ixgbe 0000:05:00.0: failed to load because an unsupported SFP+
or QSFP module type was detected.
[ 4817.038630] ixgbe 0000:05:00.0: Reload the driver after installing a
supported module.
[ 4817.418360] ixgbe 0000:41:00.1: failed to load because an unsupported SFP+
or QSFP module type was detected.
[ 4817.429354] ixgbe 0000:41:00.1: Reload the driver after installing a
supported module.
[ 4817.838086] ixgbe 0000:43:00.0: failed to load because an unsupported SFP+
or QSFP module type was detected.
[ 4817.849091] ixgbe 0000:43:00.0: Reload the driver after installing a
supported module.
[ 4817.889189] ixgbe 0000:43:00.1: failed to load because an unsupported SFP+
or QSFP module type was detected.
[ 4817.900178] ixgbe 0000:43:00.1: Reload the driver after installing a
supported module.
[ 4817.940318] ixgbe 0000:44:00.0: failed to load because an unsupported SFP+
or QSFP module type was detected.
[ 4817.951357] ixgbe 0000:44:00.0: Reload the driver after installing a
supported module.
root@ubuntu1404:~# modinfo ixgbe
filename:
/lib/modules/3.13.0-91-generic/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version: 4.3.15
license: GPL
description: Intel(R) 10 Gigabit PCI Express Network Driver
author: Intel Corporation, <linux.n...@intel.com>
srcversion: 7AED484083B2C5B86424A3A
alias: pci:v00008086d000015ADsv*sd*bc*sc*i*
alias: pci:v00008086d000015ACsv*sd*bc*sc*i*
alias: pci:v00008086d000015ABsv*sd*bc*sc*i*
alias: pci:v00008086d000015AAsv*sd*bc*sc*i*
alias: pci:v00008086d000015D1sv*sd*bc*sc*i*
alias: pci:v00008086d00001563sv*sd*bc*sc*i*
alias: pci:v00008086d00001560sv*sd*bc*sc*i*
alias: pci:v00008086d00001558sv*sd*bc*sc*i*
alias: pci:v00008086d0000154Asv*sd*bc*sc*i*
alias: pci:v00008086d00001557sv*sd*bc*sc*i*
alias: pci:v00008086d0000154Fsv*sd*bc*sc*i*
alias: pci:v00008086d0000154Dsv*sd*bc*sc*i*
alias: pci:v00008086d00001528sv*sd*bc*sc*i*
alias: pci:v00008086d000010F8sv*sd*bc*sc*i*
alias: pci:v00008086d0000151Csv*sd*bc*sc*i*
alias: pci:v00008086d00001529sv*sd*bc*sc*i*
alias: pci:v00008086d0000152Asv*sd*bc*sc*i*
alias: pci:v00008086d000010F9sv*sd*bc*sc*i*
alias: pci:v00008086d00001514sv*sd*bc*sc*i*
alias: pci:v00008086d00001507sv*sd*bc*sc*i*
alias: pci:v00008086d000010FBsv*sd*bc*sc*i*
alias: pci:v00008086d00001517sv*sd*bc*sc*i*
alias: pci:v00008086d000010FCsv*sd*bc*sc*i*
alias: pci:v00008086d000010F7sv*sd*bc*sc*i*
alias: pci:v00008086d00001508sv*sd*bc*sc*i*
alias: pci:v00008086d000010DBsv*sd*bc*sc*i*
alias: pci:v00008086d000010F4sv*sd*bc*sc*i*
alias: pci:v00008086d000010E1sv*sd*bc*sc*i*
alias: pci:v00008086d000010F1sv*sd*bc*sc*i*
alias: pci:v00008086d000010ECsv*sd*bc*sc*i*
alias: pci:v00008086d000010DDsv*sd*bc*sc*i*
alias: pci:v00008086d0000150Bsv*sd*bc*sc*i*
alias: pci:v00008086d000010C8sv*sd*bc*sc*i*
alias: pci:v00008086d000010C7sv*sd*bc*sc*i*
alias: pci:v00008086d000010C6sv*sd*bc*sc*i*
alias: pci:v00008086d000010B6sv*sd*bc*sc*i*
depends: ptp,dca,vxlan
vermagic: 3.13.0-91-generic SMP mod_unload modversions
parm: InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI,
2=MSI-X), default IntMode (deprecated) (array of int)
parm: IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X),
default 2 (array of int)
parm: MQ:Disable or enable Multiple Queues, default 1 (array of int)
parm: DCA:Disable or enable Direct Cache Access, 0=disabled,
1=descriptor only, 2=descriptor and data (array of int)
parm: RSS:Number of Receive-Side Scaling Descriptor Queues, default
0=number of cpus (array of int)
parm: VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable,
2-16 enable (default=8) (array of int)
parm: max_vfs:Number of Virtual Functions: 0 = disable (default),
1-63 = enable this many VFs (array of int)
parm: VEPA:VEPA Bridge Mode: 0 = VEB (default), 1 = VEPA (array of
int)
parm: InterruptThrottleRate:Maximum interrupts per second, per
vector, (0,1,956-488281), default 1 (array of int)
parm: LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of
int)
parm: LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array
of int)
parm: LLISize:Low Latency Interrupt on Packet Size (0-1500) (array
of int)
parm: LLIEType:Low Latency Interrupt Ethernet Protocol Type (array
of int)
parm: LLIVLANP:Low Latency Interrupt on VLAN priority threshold
(array of int)
parm: FdirPballoc:Flow Director packet buffer allocation level:
1 = 8k hash filters or 2k perfect filters
2 = 16k hash filters or 4k perfect filters
3 = 32k hash filters or 8k perfect filters (array of
int)
parm: AtrSampleRate:Software ATR Tx packet sample rate (array of
int)
parm: FCoE:Disable or enable FCoE Offload, default 1 (array of int)
parm: MDD:Malicious Driver Detection: (0,1), default 1 = on (array
of int)
parm: LRO:Large Receive Offload (0,1), default 0 = off (array of
int)
parm: allow_unsupported_sfp:Allow unsupported and untested SFP+
modules on 82599 based adapters, default 0 = Disable (array of int)
parm: dmac_watchdog:DMA coalescing watchdog in microseconds
(0,41-10000), default 0 = off (array of int)
parm: vxlan_rx:VXLAN receive checksum offload (0,1), default 1 =
Enable (array of int)
root@ubuntu1404:~# modprobe vfio-pci
00.0/sriov_numvfs
sriov_numvfssys/devices/pci0000:00/0000:00:01.0/0000:01:00.1/sriov_numvfsroot@ubuntu1404:~#
echo "1" > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/
sriov_numvfs404:~# echo "1" >
/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.1/
root@ubuntu1404:~# resize
COLUMNS=211;
LINES=53;
export COLUMNS LINES;
root@ubuntu1404:~# ip link set dev eth1 vf 0 spoofchk off
et dev eth1 vf 0 mac de:ad:be:ef:00:00
ip link set dev eth9 vf 0 mac de:ad:be:ef:00:01
root@ubuntu1404:~# ip link set dev eth9 vf 0 spoofchk off
root@ubuntu1404:~# ip link set dev eth1 vf 0 mac de:ad:be:ef:00:00
root@ubuntu1404:~# ip link set dev eth9 vf 0 mac de:ad:be:ef:00:01
root@ubuntu1404:~#
root@ubuntu1404:~# rmmod ixgbevf
root@ubuntu1404:~# modprobe ixgbevf
root@ubuntu1404:~# ip link set eth1 up
root@ubuntu1404:~# ip link set eth9 up
root@ubuntu1404:~#
root@ubuntu1404:~# bridge link set dev eth1 hwmode veb
root@ubuntu1404:~# bridge link set dev eth9 hwmode veb
root@ubuntu1404:~#
root@ubuntu1404:~# echo "8086 10ed" > /sys/bus/pci/drivers/vfio-pci/new_id
root@ubuntu1404:~# echo "0000:01:10.0" >
/sys/bus/pci/devices/0000\:01\:10.0/driver/unbind
root@ubuntu1404:~# echo "0000:01:10.1" >
/sys/bus/pci/devices/0000\:01\:10.1/driver/unbind
root@ubuntu1404:~# echo "0000:01:10.0" > /sys/bus/pci/drivers/vfio-pci/bind
root@ubuntu1404:~# echo "0000:01:10.1" > /sys/bus/pci/drivers/vfio-pci/bind
root@ubuntu1404:~#
root@ubuntu1404:~#
root@ubuntu1404:~# ethtool -l eth1
Channel parameters for eth1:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 1
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
root@ubuntu1404:~# ethtool -l eth9
Channel parameters for eth9:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 1
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
root@ubuntu1404:~# tc qdisc add dev eth1 root mqprio num_tc 8
root@ubuntu1404:~# tc qdisc add dev eth9 root mqprio num_tc 8
root@ubuntu1404:~#
root@ubuntu1404:~# taskset -c 0,2,26,4,28,6,30,8,32,10,34,12,36,14,38
qemu-system-x86_64 -enable-kvm -cpu host -smp 15 \
> -serial telnet::4444,server,nowait \
> -monitor telnet::5555,server,nowait \
> -device virtio-net,mac=DE:AD:DE:01:02:03,netdev=user.0 \
> -netdev user,id=user.0,dnssearch=vm.6wind.com,hostfwd=tcp::2222-:22 \
> -hda centos-7-template.qcow2 -m 6G \
> -device vfio-pci,host=01:10.0 -device vfio-pci,host=01:10.1 --nographic &
[1] 2851
root@ubuntu1404:~# [ 4917.696698] kvm [2851]: vcpu0 unhandled rdmsr: 0x1c9
[ 4917.702297] kvm [2851]: vcpu0 unhandled rdmsr: 0x1a6
[ 4917.707877] kvm [2851]: vcpu0 unhandled rdmsr: 0x1a7
[ 4917.713456] kvm [2851]: vcpu0 unhandled rdmsr: 0x3f6
[ 4918.731234] kvm [2851]: vcpu2 unhandled rdmsr: 0x606
[ 4922.596487] kvm [2851]: vcpu12 unhandled rdmsr: 0xd90
On the VM side
---------------
CentOS Linux 7 (Core)
Kernel 3.10.0-327.22.2.el7.x86_64 on an x86_64
centos7 login: root
Password:
Last login: Tue Sep 6 17:33:59 on ttyS0
[root@centos7 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff
3: ens5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff
4: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UP qlen 1000
link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic mgmt0
valid_lft 86256sec preferred_lft 86256sec
inet6 fe80::dcad:deff:fe01:203/64 scope link
valid_lft forever preferred_lft forever
[root@centos7 ~]# ip li set ens4 up
[ 162.228284] ixgbevf 0000:00:04.0: NIC Link is Up 10 Gbps
[root@centos7 ~]# ip li set ens5 up
[ 166.828243] ixgbevf 0000:00:05.0: NIC Link is Up 10 Gbps
[ 166.830297] IPv6: ADDRCONF(NETDEV_UP): ens5: link is not ready
[ 166.831105] IPv6: ADDRCONF(NETDEV_CHANGE): ens5: link becomes ready
[root@centos7 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen
1000
link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:feef:0/64 scope link
valid_lft forever preferred_lft forever
3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen
1000
link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:feef:1/64 scope link
valid_lft forever preferred_lft forever
4: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UP qlen 1000
link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic mgmt0
valid_lft 86238sec preferred_lft 86238sec
inet6 fe80::dcad:deff:fe01:203/64 scope link
valid_lft forever preferred_lft forever
[root@centos7 ~]# ip link add link ens4 name vlan1 type vlan id 1
[ 198.884719] 8021q: 802.1Q VLAN Support v1.8
[ 198.885423] 8021q: adding VLAN 0 to HW filter on device ens4
[ 198.886916] 8021q: adding VLAN 0 to HW filter on device ens5
[ 198.891302] 8021q: adding VLAN 0 to HW filter on device mgmt0
RTNETLINK answers: Permission denied
[root@centos7 ~]# ip link add link ens5 name vlan2 type vlan id 2
RTNETLINK answers: Permission denied
[root@centos7 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen
1000
link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:feef:0/64 scope link
valid_lft forever preferred_lft forever
3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen
1000
link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:feef:1/64 scope link
valid_lft forever preferred_lft forever
4: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UP qlen 1000
link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic mgmt0
valid_lft 86188sec preferred_lft 86188sec
inet6 fe80::dcad:deff:fe01:203/64 scope link
valid_lft forever preferred_lft forever
[root@centos7 ~]# tail -10 /var/log/messages
Sep 6 18:56:13 centos7 systemd: Started Session 1 of user root.
Sep 6 18:56:13 centos7 systemd: Starting Session 1 of user root.
Sep 6 18:56:39 centos7 kernel: ixgbevf 0000:00:04.0: NIC Link is Up 10 Gbps
Sep 6 18:56:43 centos7 kernel: ixgbevf 0000:00:05.0: NIC Link is Up 10 Gbps
Sep 6 18:56:43 centos7 kernel: IPv6: ADDRCONF(NETDEV_UP): ens5: link is not
ready
Sep 6 18:56:43 centos7 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ens5: link
becomes ready
Sep 6 18:57:15 centos7 kernel: 8021q: 802.1Q VLAN Support v1.8
Sep 6 18:57:15 centos7 kernel: 8021q: adding VLAN 0 to HW filter on device
ens4
Sep 6 18:57:15 centos7 kernel: 8021q: adding VLAN 0 to HW filter on device
ens5
Sep 6 18:57:15 centos7 kernel: 8021q: adding VLAN 0 to HW filter on device
mgmt0
[root@centos7 ~]#
Even with an ubuntu-14.04 VM, issue can be reproduced:
root@ubuntu1404:~# modinfo ixgbevf
filename:
/lib/modules/3.13.0-92-generic/kernel/drivers/net/ixgbevf/ixgbevf.ko
version: 2.16.1
license: GPL
description: Intel(R) 10 Gigabit Virtual Function Network Driver
author: Intel Corporation, <linux.n...@intel.com>
srcversion: BD04DA0615170C30891C96E
alias: pci:v00008086d00001515sv*sd*bc*sc*i*
alias: pci:v00008086d000010EDsv*sd*bc*sc*i*
depends:
vermagic: 3.13.0-92-generic SMP mod_unload modversions
parm: InterruptThrottleRate:Maximum interrupts per second, per
vector, (956-488281, 0=off, 1=dynamic), default 1 (array of int)
root@ubuntu1404:~# uname -a
Linux ubuntu1404 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC
2016 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu1404:~# lsb_release -da
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
root@ubuntu1404:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UP group default qlen 1000
link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global mgmt0
valid_lft forever preferred_lft forever
inet6 fe80::dcad:deff:fe01:203/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group
default qlen 1000
link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:feef:0/64 scope link
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group
default qlen 1000
link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:feef:1/64 scope link
valid_lft forever preferred_lft forever
root@ubuntu1404:~# ip link add link eth1 name vlan1 type vlan id 1
RTNETLINK answers: Permission denied
root@ubuntu1404:~# ip link add link eth2 name vlan2 type vlan id 2
RTNETLINK answers: Permission denied
root@ubuntu1404:~#
Thus, the VLAN creation is not supported with VF queueuing enabled (8
combined queues which is the maximum).
Yet, with default queues (without typing tc command), VLANs were properly
created.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1621474/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
