[dpdk-dev] : rx_no_dma_resources starts climbing on Intel 82599 SR-IOV ports
Hello All, Perhaps you can shade some light on an interesting problem we have. I am working with Intel 82599 NICs in a SR-IOV mode. After several hours of running traffic we checked port statistics on a host and saw that the counter rx_no_dma_resources began climbing on some ports. At that exact time the VM applications connected to that ports showed packet loss and crashed. First question, do you have a clue what is it related to? (I run the same application on all ports but not all of them experience such thing) After digging a bit I saw that the RX descriptors ring size on the host ports was defined to 256 out of 4096 possible (I used ehttool -G). I know that in DPDK I define each port's RX queue to be 256 descriptors, but this is done per VM port which means per VF of the physical NIC. * Do you know what is the relationship, if any, between the number of descriptors defined in DPDK per VM port (VF on host) using rte_eth_rx_queue_setup () and the number of descriptors defined on Host per physical port (using ethtool -g rx ) ? Seems to me that they don't effect one another but perhaps I am wrong. Thanks, Yan
[dpdk-dev] Interrupt mode in ixgbe driver and particularly in ixgbevf
Hello, I have a question about interrupt mode in ixgbe driver and particularly in ixgbevf. I am using ixgbevf driver for my ports (SR-IOV). Could you please tell me if there is any option to register with a callback on RX queue events on some port (VF) or the only option is to sleep (using rte_epoll_wait()) until an RX interrupt triggers (as suggested in examples/l3fwd-power/main.c)? I saw that there is a function rte_intr_callback_register(), but this function looks to me as relevant only for linux UIO mechanism (e.g SOCKETS, PIPE, etc ...) and not DPDK. I will welcome and appreciate any comment or relevant example on that. Thank you, Yan
[dpdk-dev] Performance issue when running DPDK on Fedora
Hi Thomas I am running a simple dpdk application on 2 VMs running on 2 different hosts. Each VM has a single port bound to a DPDK's ixgbevf module and the ports are connected to allow a traffic transaction. Both VMs are running on KVM hypervisor on Fedora. The test is quite simple, one of the VMs generates a 1500 byte packets and sends them to another VM that simply returns them back with no processing at all. For each packet I am measuring a roundtrip time for a latency analysis. According to the results I see exactly the same pattern in all the tests: 99% of the packets have a roundtrip time around 55usec when there are always several packets that have a round trip over a 1msec that is totally unacceptable in my application. Can you please advise what should be done on a DPDK / KVM / Linux basis in order to improve this result. Thanks Yan
[dpdk-dev] FW: Performance issue when running DPDK on Fedora
Hello DPDK developers, I am running a simple dpdk application on 2 VMs running on 2 different hosts. Each VM has a single port bound to a DPDK's ixgbevf module and the ports are connected to allow a traffic transaction. Both VMs are running on KVM hypervisor on Fedora. The test is quite simple, one of the VMs generates a 1500 byte packets and sends them to another VM that simply returns them back with no processing at all. For each packet I am measuring a roundtrip time for a latency analysis. According to the results I see exactly the same pattern in all the tests: 99% of the packets have a roundtrip time around 55usec when there are always several packets that have a round trip over a 1msec that is totally unacceptable in my application. Can you please advise what should be done on a DPDK / KVM / Linux basis in order to improve this result. Any suggestions will be highly appreciated. Thank you, Yan
Re: [dpdk-dev] i40eVF pmd vlan id handling
Thanks a lot, I will check and update. -Yan From: Dey, Souvik Sent: Monday, June 15, 2020 3:44 PM To: Yan Fridland Cc: dev@dpdk.org Subject: RE: i40eVF pmd vlan id handling We need to enable DEV_RX_OFFLOAD_VLAN_FILTER from the DPDK app, and then configure the specific vlan id using rte_eth_dev_vlan_filter() to have vlan id come up to the guest. By default in VmWare I guess VLAN+MAC filtering gets enabled as, we set 0/4095 vlan_id on the VF to allow all Vlans. Can you check by setting the above offload and if that helps you . -- Thanks, Souvik From: Yan Fridland mailto:y...@asocscloud.com>> Sent: Monday, June 15, 2020 8:34 AM To: Dey, Souvik mailto:so...@rbbn.com>> Subject: RE: i40eVF pmd vlan id handling NOTICE: This email was received from an EXTERNAL sender Hi Dey, As I see in my tests with i40en on VMware, I don't have any issues when working with PF directly in Passthrough mode without SR-IOV. Problems start when working with VF (i40evf) and in my case I don't even try to do any offloading, but simply send a predefined tagged packet (tagged in application) to the port and I see that it's not even leaving the port. Its stuck somewhere between VF and PF. Perhaps you know what I should configure if I don't want HW offloading in VLAN handling and I want to tag by myself .. ? Thanks Yan -Original Message- From: users mailto:users-boun...@dpdk.org>> On Behalf Of Dey, Souvik Sent: Monday, April 13, 2020 7:04 PM To: dev@dpdk.org<mailto:dev@dpdk.org>; us...@dpdk.org<mailto:us...@dpdk.org>; beilei.x...@intel.com<mailto:beilei.x...@intel.com> Cc: ferruh.yi...@intel.com<mailto:ferruh.yi...@intel.com>; qi.z.zh...@intel.com<mailto:qi.z.zh...@intel.com> Subject: [dpdk-users] i40eVF pmd vlan id handling Hi All, I am using DPDK 18.11.2 and i40e PF linux driver on the host 2.4.6. I see there, when I enable DEV_RX_OFFLOAD_VLAN_FILTER from the DPDK app, and then configure the specific vlan id using rte_eth_dev_vlan_filter(). As per DPDK code by default when we do dev_configure, we call i40evf_init_vlan() and if we don't enable rxmode.offloads with DEV_RX_OFFLOAD_VLAN_STRIP, then we should send VIRTCHNL_OP_DISABLE_VLAN_STRIPPING command to the PF from the guest. With more debugging enabled, when DPDK requests VLAN filtering by sending VIRTCHNL_OP_ADD_VLAN to the PF, then we see that the VF stripping is enabled also on Linux. If we don't add VLAN ID and send VIRTCHNL_OP_ADD_VLAN message down to the PF , then we do receive packets with VLAN ID set. The same works fine in VmWare drivers, where we do receive VALN id in the packets if STRIP is disabled. In the linux case, when we receive frames with VLAN headers, the vlan id is stripped at the PF, and the TCI will record the missing VLAN details when handed up to the DPDK driver. With i40e debug enabled, it's clear to see the difference being reported in i40e_rxd_to_vlan_tci: Example using VLAN on i40evf SR-IOV (vlan fails): PMD: i40e_rxd_to_vlan_tci(): Mbuf vlan_tci: 8, vlan_tci_outer: 0 Port 0 pkt-len=60 nb-segs=1 ETH: src=00:10:E0:8D:A7:52 dst=FF:FF:FF:FF:FF:FF type=0x0806 ARP: hrd=1 proto=0x0800 hln=6 pln=4 op=1 (ARP Request) sha=00:10:E0:8D:A7:52 sip=8.8.8.102<http://8.8.8.102> tha=00:00:00:00:00:00 tip=8.8.8.3<http://8.8.8.3> As the application requested tagging not be stripped, and the hardware driver was not able to disable strip, in my opinion either DPDK or linux driver should have a bug in handing this. Also as we see VmWare i40e driver working it could be either a extra config required for linux driver or a bug in linux driver. I also tried to add a call to rte_vlan_insert() to reinstate the VLAN header using the data from TCI in i40e_rxd_to_vlan_tci(), but it is having performance impacts as expected. Also as rte_vlan_insert() does not support QINQ, it will have other issues too. Can you please clarify if this is working for i40e vlan filter test and if we are missing something in the DPDK, like sending VIRTCHNL_OP_DISABLE_VLAN_STRIPPING after every VIRTCHNL_OP_ADD_VLAN if DEV_RX_OFFLOAD_VLAN_STRIP is not set by the DPDk app. -- Regards, Souvik Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.
Re: [dpdk-dev] i40eVF pmd vlan id handling
Actually that didnt work for me. I added both the flag and called rte_eth_dev_vlan_filter() for the rellevant port and vlan, however no packets are sent. BTW, how flag resposible for rx is relevant for my tx problem? Thanks Yan Sent from my Huawei Mobile Original Message Subject: RE: i40eVF pmd vlan id handling From: "Dey, Souvik" To: Yan Fridland CC: dev@dpdk.org We need to enable DEV_RX_OFFLOAD_VLAN_FILTER from the DPDK app, and then configure the specific vlan id using rte_eth_dev_vlan_filter() to have vlan id come up to the guest. By default in VmWare I guess VLAN+MAC filtering gets enabled as, we set 0/4095 vlan_id on the VF to allow all Vlans. Can you check by setting the above offload and if that helps you . -- Thanks, Souvik From: Yan Fridland Sent: Monday, June 15, 2020 8:34 AM To: Dey, Souvik Subject: RE: i40eVF pmd vlan id handling NOTICE: This email was received from an EXTERNAL sender Hi Dey, As I see in my tests with i40en on VMware, I don't have any issues when working with PF directly in Passthrough mode without SR-IOV. Problems start when working with VF (i40evf) and in my case I don't even try to do any offloading, but simply send a predefined tagged packet (tagged in application) to the port and I see that it's not even leaving the port. Its stuck somewhere between VF and PF. Perhaps you know what I should configure if I don't want HW offloading in VLAN handling and I want to tag by myself .. ? Thanks Yan -Original Message- From: users mailto:users-boun...@dpdk.org>> On Behalf Of Dey, Souvik Sent: Monday, April 13, 2020 7:04 PM To: dev@dpdk.org<mailto:dev@dpdk.org>; us...@dpdk.org<mailto:us...@dpdk.org>; beilei.x...@intel.com<mailto:beilei.x...@intel.com> Cc: ferruh.yi...@intel.com<mailto:ferruh.yi...@intel.com>; qi.z.zh...@intel.com<mailto:qi.z.zh...@intel.com> Subject: [dpdk-users] i40eVF pmd vlan id handling Hi All, I am using DPDK 18.11.2 and i40e PF linux driver on the host 2.4.6. I see there, when I enable DEV_RX_OFFLOAD_VLAN_FILTER from the DPDK app, and then configure the specific vlan id using rte_eth_dev_vlan_filter(). As per DPDK code by default when we do dev_configure, we call i40evf_init_vlan() and if we don't enable rxmode.offloads with DEV_RX_OFFLOAD_VLAN_STRIP, then we should send VIRTCHNL_OP_DISABLE_VLAN_STRIPPING command to the PF from the guest. With more debugging enabled, when DPDK requests VLAN filtering by sending VIRTCHNL_OP_ADD_VLAN to the PF, then we see that the VF stripping is enabled also on Linux. If we don't add VLAN ID and send VIRTCHNL_OP_ADD_VLAN message down to the PF , then we do receive packets with VLAN ID set. The same works fine in VmWare drivers, where we do receive VALN id in the packets if STRIP is disabled. In the linux case, when we receive frames with VLAN headers, the vlan id is stripped at the PF, and the TCI will record the missing VLAN details when handed up to the DPDK driver. With i40e debug enabled, it's clear to see the difference being reported in i40e_rxd_to_vlan_tci: Example using VLAN on i40evf SR-IOV (vlan fails): PMD: i40e_rxd_to_vlan_tci(): Mbuf vlan_tci: 8, vlan_tci_outer: 0 Port 0 pkt-len=60 nb-segs=1 ETH: src=00:10:E0:8D:A7:52 dst=FF:FF:FF:FF:FF:FF type=0x0806 ARP: hrd=1 proto=0x0800 hln=6 pln=4 op=1 (ARP Request) sha=00:10:E0:8D:A7:52 sip=8.8.8.102<http://8.8.8.102> tha=00:00:00:00:00:00 tip=8.8.8.3<http://8.8.8.3> As the application requested tagging not be stripped, and the hardware driver was not able to disable strip, in my opinion either DPDK or linux driver should have a bug in handing this. Also as we see VmWare i40e driver working it could be either a extra config required for linux driver or a bug in linux driver. I also tried to add a call to rte_vlan_insert() to reinstate the VLAN header using the data from TCI in i40e_rxd_to_vlan_tci(), but it is having performance impacts as expected. Also as rte_vlan_insert() does not support QINQ, it will have other issues too. Can you please clarify if this is working for i40e vlan filter test and if we are missing something in the DPDK, like sending VIRTCHNL_OP_DISABLE_VLAN_STRIPPING after every VIRTCHNL_OP_ADD_VLAN if DEV_RX_OFFLOAD_VLAN_STRIP is not set by the DPDk app. -- Regards, Souvik Notice: This e-mail together with any attachments may contain information of Ribbon Communications Inc. that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.
