create a VF with SR-IOV. VPP take over VF *directly in host.* Other hosts can not communicate with the VF in VPP because they can not get the VF neighbor. But in VPP, it can ping other hosts by VF, then the hosts get the VF neighbor. How can I make VF work like Phycics nic?
Test details as follows: *1. Startup parameters:* [root@localhost ~]# lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: CentOS Description: CentOS Linux release 7.7.1908 (Core) Release: 7.7.1908 Codename: Core [root@localhost nete]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=vg00/Root rhgb quiet intel_iommu=on iommu=pt igb.max_vfs=1" GRUB_DISABLE_RECOVERY="true" [root@localhost nete]# *2. VF interface created: * enp5s0f1 vf 0 will be used. * * [root@localhost nete]# ip link show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp5s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether a0:36:9f:09:63:9c brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off 5: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether ac:1f:6b:02:f5:c0 brd ff:ff:ff:ff:ff:ff 9: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 52:54:00:a8:d0:4f brd ff:ff:ff:ff:ff:ff 10: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT group default qlen 1000 link/ether 52:54:00:a8:d0:4f brd ff:ff:ff:ff:ff:ff 18: enp5s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether a0:36:9f:09:63:9d brd ff:ff:ff:ff:ff:ff vf 0 MAC e2:d6:86:e3:d1:17, spoof checking off, link-state auto, trust on 19: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether 00:1b:21:bb:3d:00 brd ff:ff:ff:ff:ff:ff *3. Change driver to vfio-pci:* [root@localhost ~]# lsmod | grep vfio vfio_pci 41412 0 vfio_iommu_type1 22440 0 vfio 32657 2 vfio_iommu_type1,vfio_pci irqbypass 13503 2 kvm,vfio_pci [root@localhost nete]# dpdk-devbind.py -s Network devices using DPDK-compatible driver ============================================ 0000:05:10.0 'I350 Ethernet Controller Virtual Function' drv=vfio-pci unused=igbvf,uio_pci_generic 0000:05:10.1 'I350 Ethernet Controller Virtual Function' drv=vfio-pci unused=igbvf,uio_pci_generic 0000:0b:00.0 'I210 Gigabit Network Connection' drv=vfio-pci unused=igb,uio_pci_generic Network devices using kernel driver =================================== 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp1s0 drv=ixgbe unused=vfio-pci,uio_pci_generic 0000:05:00.0 'I350 Gigabit Network Connection' if=enp5s0f0 drv=igb unused=vfio-pci,uio_pci_generic 0000:05:00.1 'I350 Gigabit Network Connection' if=enp5s0f1 drv=igb unused=vfio-pci,uio_pci_generic 0000:0a:00.0 'I210 Gigabit Network Connection' if=eno1 drv=igb unused=vfio-pci,uio_pci_generic Other network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other crypto devices ==================== <none> VPP take charge to *4. start VPP:* [root@localhost nete]# vppctl _______ _ _ _____ ___ __/ __/ _ \ (_)__ | | / / _ \/ _ \ _/ _// // / / / _ \ | |/ / ___/ ___/ /_/ /____(_)_/\___/ |___/_/ /_/ vpp# show int Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count GigabitEthernetb/0/0 3 down 9000/0/0/0 VirtualFunctionEthernet5/10/0 1 down 9000/0/0/0 VirtualFunctionEthernet5/10/1 2 up 9000/0/0/0 rx packets 1 rx bytes 60 tx packets 3 tx bytes 278 drops 1 local0 0 down 0/0/0/0 drops 1 vpp# show int addr GigabitEthernetb/0/0 (dn): VirtualFunctionEthernet5/10/0 (dn): VirtualFunctionEthernet5/10/1 (up): L3 10.2.21.210/24 L3 240e:ff:e000:c:a236:9fff:fe09:210d/64 local0 (dn): vpp# It seems VirtualFunctionEthernet5/10/1 can tx/rx packets. *5. But another host can not ping it.* [ckun@localhost ~]$ ping6 240e:ff:e000:c:a236:9fff:fe09:210d PING 240e:ff:e000:c:a236:9fff:fe09:210d(240e:ff:e000:c:a236:9fff:fe09:210d) 56 data bytes >From 240e:ff:e000:c:4735:879f:4b07:76e icmp_seq=1 Destination unreachable: >Address unreachable >From 240e:ff:e000:c:4735:879f:4b07:76e icmp_seq=2 Destination unreachable: >Address unreachable >From 240e:ff:e000:c:4735:879f:4b07:76e icmp_seq=3 Destination unreachable: >Address unreachable >From 240e:ff:e000:c:4735:879f:4b07:76e icmp_seq=4 Destination unreachable: >Address unreachable >From 240e:ff:e000:c:4735:879f:4b07:76e icmp_seq=5 Destination unreachable: >Address unreachable >From 240e:ff:e000:c:4735:879f:4b07:76e icmp_seq=6 Destination unreachable: >Address unreachable >From 240e:ff:e000:c:4735:879f:4b07:76e icmp_seq=7 Destination unreachable: >Address unreachable >From 240e:ff:e000:c:4735:879f:4b07:76e icmp_seq=8 Destination unreachable: >Address unreachable [ckun@localhost ~]$ ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 88.88.88.110 netmask 255.255.255.0 broadcast 88.88.88.255 inet6 240e:ff:e000:c:4735:879f:4b07:76e prefixlen 64 scopeid 0x0<global> inet6 fe80::80aa:89d7:b5dd:dbb8 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:6a:73:24 txqueuelen 1000 (Ethernet) RX packets 40150 bytes 3530584 (3.3 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 7000 bytes 463269 (452.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 18 memory 0xfd4a0000-fd4c0000 Trace packets in VPP: No icmpv6 packet, no ns/nd packet. Only ipv4 arp packets. *6. The host can not find the VF neighbor.* [ckun@localhost ~]$ ip -6 neigh fe80::5a6a:b1ff:fe74:787a dev eth1 lladdr 58:6a:b1:74:78:7a router STALE fe80::5a6a:b1ff:fe74:787a dev eth0 lladdr 58:6a:b1:74:78:7a router STALE 240e:ff:e000:c:a236:9fff:fe09:210d dev eth0 FAILED *7. Then ping the host in VPP, it works* vpp# ping 240e:ff:e000:c:4735:879f:4b07:76e 76 bytes from 240e:ff:e000:c:4735:879f:4b07:76e: icmp_seq=2 ttl=64 time=.6753 ms 76 bytes from 240e:ff:e000:c:4735:879f:4b07:76e: icmp_seq=3 ttl=64 time=.6597 ms 76 bytes from 240e:ff:e000:c:4735:879f:4b07:76e: icmp_seq=4 ttl=64 time=.6401 ms 76 bytes from 240e:ff:e000:c:4735:879f:4b07:76e: icmp_seq=5 ttl=64 time=.6447 ms Check neighbor, the host got the VF mac [ckun@localhost ~]$ ip -6 neigh fe80::5a6a:b1ff:fe74:787a dev eth1 lladdr 58:6a:b1:74:78:7a router STALE fe80::5a6a:b1ff:fe74:787a dev eth0 lladdr 58:6a:b1:74:78:7a router STALE 240e:ff:e000:c:a236:9fff:fe09:210d dev eth0 lladdr ee:62:d7:6c:3c:bc REACHABLE *8.The host can ping VF, too* [ckun@localhost ~]$ ping6 240e:ff:e000:c:a236:9fff:fe09:210d PING 240e:ff:e000:c:a236:9fff:fe09:210d(240e:ff:e000:c:a236:9fff:fe09:210d) 56 data bytes 64 bytes from 240e:ff:e000:c:a236:9fff:fe09:210d: icmp_seq=1 ttl=63 time=0.471 ms 64 bytes from 240e:ff:e000:c:a236:9fff:fe09:210d: icmp_seq=2 ttl=63 time=0.505 ms 64 bytes from 240e:ff:e000:c:a236:9fff:fe09:210d: icmp_seq=3 ttl=63 time=0.448 ms ^C --- 240e:ff:e000:c:a236:9fff:fe09:210d ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.448/0.474/0.505/0.034 ms [ckun@localhost ~]$ *9. route info, version 19.08.1* vpp# show ip6 fib ipv6-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] locks:[src:plugin-hi:1, src:recursive-resolution:1, src:default-route:1, ] ::/0 unicast-ip6-chain [@0]: dpo-load-balance: [proto:ip6 index:6 buckets:1 uRPF:25 to:[0:0]] [0] [@19]: dpo-load-balance: [proto:ip6 index:19 buckets:1 uRPF:22 to:[0:0]] [0] [@5]: ipv6 via 240e:ff:e000:c::1 VirtualFunctionEthernet5/10/1: mtu:9000 586ab174787aee62d76c3cbc86dd 240e:ff:e000:c::/64 unicast-ip6-chain [@0]: dpo-load-balance: [proto:ip6 index:17 buckets:1 uRPF:23 to:[1:116]] [0] [@4]: ipv6-glean: VirtualFunctionEthernet5/10/1: mtu:9000 ffffffffffffee62d76c3cbc86dd 240e:ff:e000:c::1/128 unicast-ip6-chain [@0]: dpo-load-balance: [proto:ip6 index:19 buckets:1 uRPF:22 to:[0:0]] [0] [@5]: ipv6 via 240e:ff:e000:c::1 VirtualFunctionEthernet5/10/1: mtu:9000 586ab174787aee62d76c3cbc86dd 240e:ff:e000:c:4735:879f:4b07:76e/128 unicast-ip6-chain [@0]: dpo-load-balance: [proto:ip6 index:27 buckets:1 uRPF:33 to:[7:776]] [0] [@5]: ipv6 via 240e:ff:e000:c:4735:879f:4b07:76e VirtualFunctionEthernet5/10/1: mtu:9000 000c296a7324ee62d76c3cbc86dd 240e:ff:e000:c:4c60:6db3:5ef0:5b6d/128 unicast-ip6-chain [@0]: dpo-load-balance: [proto:ip6 index:23 buckets:1 uRPF:29 to:[0:0]] [0] [@5]: ipv6 via 240e:ff:e000:c:4c60:6db3:5ef0:5b6d VirtualFunctionEthernet5/10/1: mtu:9000 0c9d928165cdee62d76c3cbc86dd 240e:ff:e000:c:6c89:a5ab:45b0:16da/128 unicast-ip6-chain [@0]: dpo-load-balance: [proto:ip6 index:20 buckets:1 uRPF:26 to:[0:0]] [0] [@5]: ipv6 via 240e:ff:e000:c:6c89:a5ab:45b0:16da VirtualFunctionEthernet5/10/1: mtu:9000 4cedfb463cbcee62d76c3cbc86dd 240e:ff:e000:c:85e0:1fe7:b7e:9af8/128 unicast-ip6-chain [@0]: dpo-load-balance: [proto:ip6 index:22 buckets:1 uRPF:28 to:[0:0]] [0] [@5]: ipv6 via 240e:ff:e000:c:85e0:1fe7:b7e:9af8 VirtualFunctionEthernet5/10/1: mtu:9000 0c9d928165cdee62d76c3cbc86dd 240e:ff:e000:c:a236:9fff:fe09:210d/128 unicast-ip6-chain [@0]: dpo-load-balance: [proto:ip6 index:18 buckets:1 uRPF:24 to:[9:920]] [0] [@2]: dpo-receive: 240e:ff:e000:c:a236:9fff:fe09:210d on VirtualFunctionEthernet5/10/1 fe80::/10 unicast-ip6-chain [@0]: dpo-load-balance: [proto:ip6 index:7 buckets:1 uRPF:6 to:[0:0]] [0] [@14]: ip6-link-local vpp# show version vpp v19.08.1-185~gd294b98 built by ckun on localhost.localdomain at Tue Oct 22 12:22:21 CST 2019 vpp# show hardware-interfaces Name Idx Link Hardware GigabitEthernetb/0/0 3 up GigabitEthernetb/0/0 Link speed: 1 Gbps Ethernet address ac:1f:6b:02:f5:c1 Intel e1000 carrier up full duplex mtu 9206 flags: pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum rx: queues 1 (max 4), desc 1024 (min 32 max 4096 align 8) tx: queues 1 (max 4), desc 1024 (min 32 max 4096 align 8) pci: device 8086:1533 subsystem 15d9:1533 address 0000:0b:00.00 numa 0 max rx packet len: 16383 promiscuous: unicast off all-multicast off vlan offload: strip off filter off qinq off rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum vlan-filter jumbo-frame scatter keep-crc rx offload active: ipv4-cksum jumbo-frame scatter tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum tcp-tso multi-segs tx offload active: udp-cksum tcp-cksum multi-segs rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp-ex ipv6-udp-ex ipv6-tcp ipv6-udp ipv6-ex ipv6 rss active: none tx burst function: eth_igb_xmit_pkts rx burst function: eth_igb_recv_pkts VirtualFunctionEthernet5/10/0 1 down VirtualFunctionEthernet5/10/0 Link speed: unknown Ethernet address 2e:79:be:44:96:6e Intel e1000 VF carrier down flags: pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum rx: queues 1 (max 1), desc 1024 (min 32 max 4096 align 8) tx: queues 1 (max 1), desc 1024 (min 32 max 4096 align 8) pci: device 8086:1520 subsystem 8086:0002 address 0000:05:10.00 numa 0 max rx packet len: 16383 promiscuous: unicast off all-multicast off vlan offload: strip off filter off qinq off rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum vlan-filter jumbo-frame scatter keep-crc rx offload active: ipv4-cksum jumbo-frame scatter tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum tcp-tso multi-segs tx offload active: udp-cksum tcp-cksum multi-segs rss avail: none rss active: none tx burst function: eth_igb_xmit_pkts rx burst function: eth_igb_recv_pkts VirtualFunctionEthernet5/10/1 2 up VirtualFunctionEthernet5/10/1 Link speed: 100 Mbps Ethernet address ee:62:d7:6c:3c:bc Intel e1000 VF carrier up full duplex mtu 9206 flags: admin-up pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum rx: queues 1 (max 1), desc 1024 (min 32 max 4096 align 8) tx: queues 1 (max 1), desc 1024 (min 32 max 4096 align 8) pci: device 8086:1520 subsystem 8086:0002 address 0000:05:10.01 numa 0 max rx packet len: 16383 promiscuous: unicast off all-multicast on vlan offload: strip off filter off qinq off rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum vlan-filter jumbo-frame scatter keep-crc rx offload active: ipv4-cksum jumbo-frame scatter tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum tcp-tso multi-segs tx offload active: udp-cksum tcp-cksum multi-segs rss avail: none rss active: none tx burst function: eth_igb_xmit_pkts rx burst function: eth_igb_recv_scattered_pkts tx frames ok 56 tx bytes ok 2658 rx frames ok 1434 rx bytes ok 58441 extended stats: rx good packets 1434 tx good packets 56 rx good bytes 58441 tx good bytes 2658 tx good loopback packets 7 tx good loopback bytes 218 local0 0 down local0 Link speed: unknown local vpp# vpp# vpp# What can I do with the configurations of SR-IOV or driver or something else to solve this question?
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14877): https://lists.fd.io/g/vpp-dev/message/14877 Mute This Topic: https://lists.fd.io/mt/68270891/21656 Mute #vpp: https://lists.fd.io/mk?hashtag=vpp&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
