> On 21 Feb 2020, at 10:31, chetan bhasin <chetan.bhasin...@gmail.com> wrote: > > Hi Nitin,Damjan, > > For 40G XL710 buffers : 537600 (500K+) > 1) vpp 19.08 (sept 2019) : it worked with vpp 19.08 (sept release) after > removing intel_iommu=on from Grub params. > 2) stable/vpp2001(latest) : It worked even we have "intel_iommu=on" in Grub > params > > > On stable/vpp2001 , I found a check-in before which it did not work with " > intel_iommu=on " as grub params, but after the below change-list it work even > with grub params. > commit 45495480c8165090722389b08075df06ccfcd7ef > Author: Yulong Pei <yulong....@intel.com <mailto:yulong....@intel.com>> > Date: Thu Oct 17 18:41:52 2019 +0800 > vlib: linux: fix wrong iommu_group value issue when using dpdk-plugin > > Before above change in vpp 20.01 , when we bring up vpp with vfio-pci, vpp > change /sys/module/vfio/parameters/enable_unsafe_noiommu_mode to "Y" , and > we face issue with traffic but after the change sys file value remain as > "N" in /sys/module/vfio/parameters/enable_unsafe_noiommu_mode and traffic > works fine. > > As it is bare metal so we can remove intel_iommu=on from grub to make it work > without any patches . Any suggestions?
IOMMU gives you following: - protection and security - it prevents misbehaving NIC to read/write intentionally or unintentionally memory it is not supposed to access - VA -> PA translation If you are running bare-metal, single tenant security is probably not concern, but still it can protect NIC from doing something bad eventually because of driver issues. VA -> PA translation helps with performance, as driver doesn’t need to lookup for PA when submitting descriptors but this is not critical perf issue. So it is up to you to decide, work without IOMMU or patch your old VPP version…. > > Regards, > Chetan > > On Tue, Feb 18, 2020 at 1:07 PM Nitin Saxena <nsax...@marvell.com > <mailto:nsax...@marvell.com>> wrote: > HI Chethan, > > > > Your packet trace shows that packet data is all 0 and that’s why you are > running into l3 mac mismatch. > > I am guessing something messed with IOMMU due to which translation is not > happening. Although packet length is correct. > > You can try out AVF plugin to iron out where problem exists, in dpdk plugin > or vlib > > > > Thanks, > > Nitin > > > > From: chetan bhasin <chetan.bhasin...@gmail.com > <mailto:chetan.bhasin...@gmail.com>> > Sent: Tuesday, February 18, 2020 12:50 PM > To: me <chetan.bhasin...@gmail.com <mailto:chetan.bhasin...@gmail.com>> > Cc: Nitin Saxena <nsax...@marvell.com <mailto:nsax...@marvell.com>>; vpp-dev > <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>> > Subject: Re: [EXT] [vpp-dev] Regarding buffers-per-numa parameter > > > > Hi, > > One more finding related to intel nic and number of buffers (537600) > > > > vpp branch > > driver > > card > > buffers > > Traffic > > Err > > stable/1908 > > uio_pci_genric > > X722(10G) > > 537600 > > Working > > > > stable/1908 > > vfio-pci > > XL710(40G) > > 537600 > > Not Working > > l3 mac mismatch > > stable/2001 > > uio_pci_genric > > X722(10G) > > 537600 > > Working > > > > stable/2001 > > vfio-pci > > XL710(40G) > > 537600 > > Working > > > > > > > > Thanks, > > Chetan > > > > On Mon, Feb 17, 2020 at 7:17 PM chetan bhasin via Lists.Fd.Io > <https://urldefense.proofpoint.com/v2/url?u=http-3A__Lists.Fd.Io&d=DwMFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=S4H7jibYAtA5YOvfL3IkGduCfk9LbZMPOAecQGDzWV0&m=qxJrqbz5sNlCrzJTOZjaJ0jHeaW077bX6ZxmV308jfg&s=ffS1Y8GHllzjueMUVW31gwrVEIK1HVSNTKk2yA-VjG8&e=> > <chetan.bhasin017=gmail....@lists.fd.io <mailto:gmail....@lists.fd.io>> > wrote: > > Hi Nitin, > > > > https://github.com/FDio/vpp/commits/stable/2001/src/vlib > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_FDio_vpp_commits_stable_2001_src_vlib&d=DwMFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=S4H7jibYAtA5YOvfL3IkGduCfk9LbZMPOAecQGDzWV0&m=qxJrqbz5sNlCrzJTOZjaJ0jHeaW077bX6ZxmV308jfg&s=LljqKCmXwjl4uzuLM_oB-jhjYV5xVGFpHPDomTZwKAU&e=> > As per stable/2001 branch , the given change is checked-in around Oct 28 2019. > > > > df0191ead2cf39611714b6603cdc5bdddc445b57 is previous commit of > b6e8b1a7c8bf9f9fbd05cdc3c90111d9e7a6897? > Yes (branch vpp 20.01) > > > > Thanks, > > Chetan Bhasin > > > > On Mon, Feb 17, 2020 at 5:33 PM Nitin Saxena <nsax...@marvell.com > <mailto:nsax...@marvell.com>> wrote: > > Hi Damjan, > > >> if you read Chetan’s email bellow, you will see that this one is already > >> excluded… > Sorry I missed that part. After seeing diffs between stable/1908 and > stable/2001, commit: b6e8b1a7c8bf9f9fbd05cdc3c90111d9e7a6897 is the only > visible git commit in dpdk plugin which is playing with mempool buffers. If > it does not solve the problem then I suspect problem lies outside dpdk > plugin. I am guessing DPDK-19.08 is being used here with VPP-19.08 > > Hi Chetan, > > > 3) I took previous commit of "vlib: don't use vector for keeping buffer > > indices in the pool " ie "df0191ead2cf39611714b6603cdc5bdddc445b57" : > > Everything looks fine with Buffers 537600. > In which branch, Commit: df0191ead2cf39611714b6603cdc5bdddc445b57 is previous > commit of b6e8b1a7c8bf9f9fbd05cdc3c90111d9e7a6897? > > Thanks, > Nitin > > -----Original Message----- > > From: Damjan Marion <dmar...@me.com <mailto:dmar...@me.com>> > > Sent: Monday, February 17, 2020 3:47 PM > > To: Nitin Saxena <nsax...@marvell.com <mailto:nsax...@marvell.com>> > > Cc: chetan bhasin <chetan.bhasin...@gmail.com > > <mailto:chetan.bhasin...@gmail.com>>; vpp-dev@lists.fd.io > > <mailto:vpp-dev@lists.fd.io> > > Subject: Re: [EXT] [vpp-dev] Regarding buffers-per-numa parameter > > > > > > Dear Nitin, > > > > if you read Chetan’s email bellow, you will see that this one is already > > excluded… > > > > Also, it will not be easy to explain how this patch blows tx function in > > dpdk > > mlx5 pmd… > > > > — > > Damjan > > > > > On 17 Feb 2020, at 11:12, Nitin Saxena <nsax...@marvell.com > > > <mailto:nsax...@marvell.com>> wrote: > > > > > > Hi Prashant/Chetan, > > > > > > I would try following change first to solve the problem in 1908 > > > > > > commit b6e8b1a7c8bf9f9fbd05cdc3c90111d9e7a6897b > > > Author: Damjan Marion <damar...@cisco.com <mailto:damar...@cisco.com>> > > > Date: Tue Mar 12 18:14:15 2019 +0100 > > > > > > vlib: don't use vector for keeping buffer indices in > > > > > > Type: refactor > > > > > > Change-Id: I72221b97d7e0bf5c93e20bbda4473ca67bfcdeb4 > > > Signed-off-by: Damjan Marion damar...@cisco.com > > > <mailto:damar...@cisco.com> > > > > > > You can also try copying src/plugins/dpdk/buffer.c from stable/2001 > > branch to stable/1908 > > > > > > Thanks, > > > Nitin > > > > > > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > > > <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>> On Behalf Of Damjan > > Marion via Lists.Fd.Io > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__Lists.Fd.Io&d=DwMFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=S4H7jibYAtA5YOvfL3IkGduCfk9LbZMPOAecQGDzWV0&m=qxJrqbz5sNlCrzJTOZjaJ0jHeaW077bX6ZxmV308jfg&s=ffS1Y8GHllzjueMUVW31gwrVEIK1HVSNTKk2yA-VjG8&e=> > > > Sent: Monday, February 17, 2020 1:52 PM > > > To: chetan bhasin <chetan.bhasin...@gmail.com > > > <mailto:chetan.bhasin...@gmail.com>> > > > Cc: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > > > Subject: [EXT] Re: [vpp-dev] Regarding buffers-per-numa parameter > > > > > > External Email > > > > > > On 17 Feb 2020, at 07:37, chetan bhasin <chetan.bhasin...@gmail.com > > > <mailto:chetan.bhasin...@gmail.com>> > > wrote: > > > > > > Bottom line is stable/vpp 908 does not work with higher number of buffers > > but stable/vpp2001 does. Could you please advise which area we can look at > > ,as it would be difficult for us to move to vpp2001 at this time. > > > > > > I really don’t have idea what caused this problem to disappear. > > > You may try to use “git bisect” to find out which commit fixed it…. > > > > > > — > > > Damjan > > > > > > > > > > > > On Mon, Feb 17, 2020 at 11:01 AM chetan bhasin via Lists.Fd.Io > > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__Lists.Fd.Io&d=DwMFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=S4H7jibYAtA5YOvfL3IkGduCfk9LbZMPOAecQGDzWV0&m=qxJrqbz5sNlCrzJTOZjaJ0jHeaW077bX6ZxmV308jfg&s=ffS1Y8GHllzjueMUVW31gwrVEIK1HVSNTKk2yA-VjG8&e=> > > <chetan.bhasin017=gmail....@lists.fd.io <mailto:gmail....@lists.fd.io>> > > wrote: > > > Thanks Damjan for the reply! > > > > > > Following are my observations on Intel X710/XL710 pci- > > > 1) I took latest code base from stable/vpp19.08 : Seeing error as " > > ethernet-input l3 mac mismatch" > > > With Buffers 537600 > > > vpp# show buffers > > | > > > Pool Name Index NUMA Size Data Size Total Avail Cached > > > Used > > > default-numa-0 0 0 2496 2048 537600 510464 1319 > > > 25817 > > > default-numa-1 1 1 2496 2048 537600 528896 390 > > > 8314 > > > > > > vpp# show hardware-interfaces > > > Name Idx Link Hardware > > > BondEthernet0 3 up BondEthernet0 > > > Link speed: unknown > > > Ethernet address 3c:fd:fe:b5:5e:40 > > > FortyGigabitEthernet12/0/0 1 up FortyGigabitEthernet12/0/0 > > > Link speed: 40 Gbps > > > Ethernet address 3c:fd:fe:b5:5e:40 > > > Intel X710/XL710 Family > > > carrier up full duplex mtu 9206 > > > flags: admin-up pmd rx-ip4-cksum > > > rx: queues 16 (max 320), desc 1024 (min 64 max 4096 align 32) > > > tx: queues 16 (max 320), desc 4096 (min 64 max 4096 align 32) > > > pci: device 8086:1583 subsystem 8086:0001 address 0000:12:00.00 numa > > 0 > > > max rx packet len: 9728 > > > promiscuous: unicast off all-multicast on > > > vlan offload: strip off filter off qinq off > > > rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum > > > qinq-strip > > > outer-ipv4-cksum vlan-filter vlan-extend > > > jumbo-frame > > > scatter keep-crc > > > rx offload active: ipv4-cksum > > > tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum > > > sctp-cksum > > > tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso > > > gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs > > > mbuf-fast-free > > > tx offload active: none > > > rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other > > > ipv6-frag > > > ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload > > > rss active: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv6-frag > > > ipv6-tcp > > > ipv6-udp ipv6-other > > > tx burst function: i40e_xmit_pkts_vec_avx2 > > > rx burst function: i40e_recv_pkts_vec_avx2 > > > tx errors 17 > > > rx frames ok 4585 > > > rx bytes ok 391078 > > > extended stats: > > > rx good packets 4585 > > > rx good bytes 391078 > > > tx errors 17 > > > rx multicast packets 4345 > > > rx broadcast packets 243 > > > rx unknown protocol packets 4588 > > > rx size 65 to 127 packets 4529 > > > rx size 128 to 255 packets 32 > > > rx size 256 to 511 packets 26 > > > rx size 1024 to 1522 packets 1 > > > tx size 65 to 127 packets 33 > > > FortyGigabitEthernet12/0/1 2 up FortyGigabitEthernet12/0/1 > > > Link speed: 40 Gbps > > > Ethernet address 3c:fd:fe:b5:5e:40 > > > Intel X710/XL710 Family > > > carrier up full duplex mtu 9206 > > > flags: admin-up pmd rx-ip4-cksum > > > rx: queues 16 (max 320), desc 1024 (min 64 max 4096 align 32) > > > tx: queues 16 (max 320), desc 4096 (min 64 max 4096 align 32) > > > pci: device 8086:1583 subsystem 8086:0000 address 0000:12:00.01 numa > > 0 > > > max rx packet len: 9728 > > > promiscuous: unicast off all-multicast on > > > vlan offload: strip off filter off qinq off > > > rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum > > > qinq-strip > > > outer-ipv4-cksum vlan-filter vlan-extend > > > jumbo-frame > > > scatter keep-crc > > > rx offload active: ipv4-cksum > > > tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum > > > sctp-cksum > > > tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso > > > gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs > > > mbuf-fast-free > > > tx offload active: none > > > rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other > > > ipv6-frag > > > ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload > > > rss active: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv6-frag > > > ipv6-tcp > > > ipv6-udp ipv6-other > > > tx burst function: i40e_xmit_pkts_vec_avx2 > > > rx burst function: i40e_recv_pkts_vec_avx2 > > > rx frames ok 4585 > > > rx bytes ok 391078 > > > extended stats: > > > rx good packets 4585 > > > rx good bytes 391078 > > > rx multicast packets 4344 > > > rx broadcast packets 243 > > > rx unknown protocol packets 4587 > > | > > > rx size 65 to 127 packets 4528 > > > rx size 128 to 255 packets 32 > > > rx size 256 to 511 packets 26 > > > rx size 1024 to 1522 packets 1 > > > tx size 65 to 127 packets 33 > > > > > > > > > As per packet trace - > > > Packet 4 > > > 00:00:54:955863: dpdk-input > > > FortyGigabitEthernet12/0/0 rx queue 0 > > > buffer 0x13fc728: current data 0, length 68, buffer-pool 0, ref-count 1, > > totlen-nifb 0, trace handle 0x1000003 > > > ext-hdr-valid > > | > > > l4-cksum-computed l4-cksum-correct > > > PKT MBUF: port 0, nb_segs 1, pkt_len 68 > > > buf_len 2176, data_len 68, ol_flags 0x180, data_off 128, phys_addr > > 0xde91ca80 > > > packet_type 0x1 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 > > > rss 0x0 fdir.hi 0x0 fdir.lo 0x0 > > > Packet Offload Flags > > > PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid > > > PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid > > > Packet Types > > > RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet > > > 0x0000: 00:00:00:00:00:00 -> 00:00:00:00:00:00 > > > 00:00:54:955864: bond-input > > > src 00:00:00:00:00:00, dst 00:00:00:00:00:00, > > > FortyGigabitEthernet12/0/0 - > > > BondEthernet0 > > > 00:00:54:955864: ethernet-input > > > 0x0000: 00:00:00:00:00:00 -> 00:00:00:00:00:00 > > > 00:00:54:955865: error-drop > > > rx:BondEthernet0 > > > 00:00:54:955865: drop > > > ethernet-input: l3 mac mismatch > > > > > > 2) I have took latest code-base from stable/vpp2001 branch: Everything > > looks fine with Buffers 537600 > > > > > > 3) I took previous commit of "vlib: don't use vector for keeping buffer > > indices in the pool " ie "df0191ead2cf39611714b6603cdc5bdddc445b57" : > > Everything looks fine with Buffers 537600. > > > So this cleary shows the above commit will not fix our problem. > > > > > > > > > > > > Thanks, > > > Chetan > > > > > > On Wed, Feb 12, 2020 at 9:07 PM Damjan Marion <dmar...@me.com > > > <mailto:dmar...@me.com>> > > wrote: > > > > > > Shouldn’t be too hard to checkout commit prior to that one and test if > > problem is still there… > > > > > > — > > > Damjan > > > > > > > > > > > > On 12 Feb 2020, at 14:50, chetan bhasin <chetan.bhasin...@gmail.com > > > <mailto:chetan.bhasin...@gmail.com>> > > wrote: > > > > > > Hi, > > > > > > Looking into the changes in vpp 20.1 , the below change looks good > > important related to buffer indices . > > > > > > vlib: don't use vector for keeping buffer indices in the pool > > > Type: refactor > > > > > > Change-Id: I72221b97d7e0bf5c93e20bbda4473ca67bfcdeb4 > > > Signed-off-by: Damjan Marion <damar...@cisco.com > > > <mailto:damar...@cisco.com>> > > > > > > https://urldefense.proofpoint.com/v2/url?u=https- > > > <https://urldefense.proofpoint.com/v2/url?u=https-> > > 3A__github.com_FDio_vpp_commit_b6e8b1a7c8bf9f9fbd05cdc3c90111d9e7 > > a6897b-23diff- > > 2D2260a8080303fbcc30ef32f782b4d6df&d=DwIFaQ&c=nKjWec2b6R0mOyPaz > > 7xtfQ&r=S4H7jibYAtA5YOvfL3IkGduCfk9LbZMPOAecQGDzWV0&m=IYJSlvQW > > nHZRFb7PgVq0RR9rayZkIR_eLIsg4QLU3VU&s=82mrobM4Iis3mDVPbnr526Wv > > 1yxa4TtVoa-WH8oCguI&e= > > > > > > Can anybody suggest ? > > > Shouldn’t be too hard to checkout commit prior to that one and test if > > problem is still there… > > > > > > — > > > Damjan > > > > > > > > > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15487): https://lists.fd.io/g/vpp-dev/message/15487 Mute This Topic: https://lists.fd.io/mt/71346533/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
