Thank you for reply kevin, I do have a multi queue configuration, as you can see following. Let me tell you what tools I am using for traffic generation. We run VoIP services (pretty much telco style) so we need low latency for the network to provide quality audio experience for customers. We developed an in-house load-testing tool which emulates voip calls and sends audio traffic and returns stats like drop/jitters etc, In short audio RTP protocol is UDP based 150 bytes packets and they hit pretty hard to the CPU. when i am running same loadtest on SRIOV vm i am seeing very super and neat audio quality without any single packet drop and I can load 20k people on a single 8 vCPU core VM (that is my baseline). similar test when i am running on DPDK based Host machine running vm and seeing very high latency and choppy audio quality.
When my PMD hits 70% and above the processing cycle then i start seeing packet drops not during pinging VM and audio quality degraded that means somewhere i am hitting limit. Notes: my VM isn't running any DPDK aware application so do you think my VM is bottleneck here? I heard that when you are using an OVS-DPDK based compute then you need your guest VM also aware of dpdk. [root@vm0 ~]# ethtool -i eth0 driver: virtio_net version: 1.0.0 firmware-version: expansion-rom-version: bus-info: 0000:00:03.0 supports-statistics: no supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no [root@vm0~]# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 8 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 8 On Mon, Nov 1, 2021 at 1:57 PM Kevin Traynor <[email protected]> wrote: > > On 30/10/2021 06:07, Satish Patel wrote: > > Folks, > > > > I have configured ovs-dpdk to replace sriov deployment for bonding > > support. everything good but somehow as soon as i start hitting > > 200kpps rate i start seeing packet drop. > > > > I have configured CPU isolation as per documentation to assign a > > dedicated pmd thread. I have assigned 8 dedicated PMD threads but > > still performance is very poor. > > > > I created an 8vCPU vm on openstack using dpdk and running some > > workload using an in-house application, during the 200kpps packet rate > > I noticed my all PMD cpu showing high CPU processing cycles. > > > > Not sure what you class as 'high CPU processing cycles', but if you mean > at the point of hitting a CPU bottleneck and the stats below are taken > with contrant traffic, none of them would indicate that is the case. > > > $ ovs-vswitchd -V > > ovs-vswitchd (Open vSwitch) 2.13.3 > > DPDK 19.11.7 > > > > In the following output what does these queue-id:0 to 8 and why only > > the first queue is in use but not others, they are always zero. What > > does this mean? > > > > It means the application is only sending packets to OVS on a single > queue. It may be that the application does not do multi-queue, or > because the application sends on the same queue number that it receives > traffic on and it is only receiving traffic on one queue. > > > ovs-appctl dpif-netdev/pmd-rxq-show > > pmd thread numa_id 0 core_id 2: > > isolated : false > > port: vhu1c3bf17a-01 queue-id: 0 (enabled) pmd usage: 0 % > > port: vhu1c3bf17a-01 queue-id: 1 (enabled) pmd usage: 0 % > > port: vhu6b7daba9-1a queue-id: 2 (disabled) pmd usage: 0 % > > port: vhu6b7daba9-1a queue-id: 3 (disabled) pmd usage: 0 % > > pmd thread numa_id 1 core_id 3: > > isolated : false > > pmd thread numa_id 0 core_id 22: > > isolated : false > > port: vhu1c3bf17a-01 queue-id: 3 (enabled) pmd usage: 0 % > > port: vhu1c3bf17a-01 queue-id: 6 (enabled) pmd usage: 0 % > > port: vhu6b7daba9-1a queue-id: 0 (enabled) pmd usage: 54 % > > port: vhu6b7daba9-1a queue-id: 5 (disabled) pmd usage: 0 % > > pmd thread numa_id 1 core_id 23: > > isolated : false > > port: dpdk1 queue-id: 0 (enabled) pmd usage: 3 % > > pmd thread numa_id 0 core_id 26: > > isolated : false > > port: vhu1c3bf17a-01 queue-id: 2 (enabled) pmd usage: 0 % > > port: vhu1c3bf17a-01 queue-id: 7 (enabled) pmd usage: 0 % > > port: vhu6b7daba9-1a queue-id: 1 (disabled) pmd usage: 0 % > > port: vhu6b7daba9-1a queue-id: 4 (disabled) pmd usage: 0 % > > pmd thread numa_id 1 core_id 27: > > isolated : false > > pmd thread numa_id 0 core_id 46: > > isolated : false > > port: dpdk0 queue-id: 0 (enabled) pmd usage: 27 % > > port: vhu1c3bf17a-01 queue-id: 4 (enabled) pmd usage: 0 % > > port: vhu1c3bf17a-01 queue-id: 5 (enabled) pmd usage: 0 % > > port: vhu6b7daba9-1a queue-id: 6 (disabled) pmd usage: 0 % > > port: vhu6b7daba9-1a queue-id: 7 (disabled) pmd usage: 0 % > > pmd thread numa_id 1 core_id 47: > > isolated : false > > > > > > $ ovs-appctl dpif-netdev/pmd-stats-clear && sleep 10 && ovs-appctl > > dpif-netdev/pmd-stats-show | grep "processing cycles:" > > processing cycles: 1697952 (0.01%) > > processing cycles: 12726856558 (74.96%) > > processing cycles: 4259431602 (19.40%) > > processing cycles: 512666 (0.00%) > > processing cycles: 6324848608 (37.81%) > > > > Does processing cycles mean my PMD is under stress? but i am only > > hitting 200kpps rate? > > > > > > This is my dpdk0 and dpdk1 port statistics > > > > sudo ovs-vsctl get Interface dpdk0 statistics > > {flow_director_filter_add_errors=153605, > > flow_director_filter_remove_errors=30829, mac_local_errors=0, > > mac_remote_errors=0, ovs_rx_qos_drops=0, ovs_tx_failure_drops=0, > > ovs_tx_invalid_hwol_drops=0, ovs_tx_mtu_exceeded_drops=0, > > ovs_tx_qos_drops=0, rx_128_to_255_packets=64338613, > > rx_1_to_64_packets=367, rx_256_to_511_packets=116298, > > rx_512_to_1023_packets=31264, rx_65_to_127_packets=6990079, > > rx_broadcast_packets=0, rx_bytes=12124930385, rx_crc_errors=0, > > rx_dropped=0, rx_errors=12, rx_fcoe_crc_errors=0, rx_fcoe_dropped=12, > > rx_fcoe_mbuf_allocation_errors=0, rx_fragment_errors=367, > > rx_illegal_byte_errors=0, rx_jabber_errors=0, rx_length_errors=0, > > rx_mac_short_packet_dropped=128, rx_management_dropped=35741, > > rx_management_packets=31264, rx_mbuf_allocation_errors=0, > > rx_missed_errors=0, rx_oversize_errors=0, rx_packets=71512362, > > rx_priority0_dropped=0, rx_priority0_mbuf_allocation_errors=1096, > > rx_priority1_dropped=0, rx_priority1_mbuf_allocation_errors=0, > > rx_priority2_dropped=0, rx_priority2_mbuf_allocation_errors=0, > > rx_priority3_dropped=0, rx_priority3_mbuf_allocation_errors=0, > > rx_priority4_dropped=0, rx_priority4_mbuf_allocation_errors=0, > > rx_priority5_dropped=0, rx_priority5_mbuf_allocation_errors=0, > > rx_priority6_dropped=0, rx_priority6_mbuf_allocation_errors=0, > > rx_priority7_dropped=0, rx_priority7_mbuf_allocation_errors=0, > > rx_undersize_errors=6990079, tx_128_to_255_packets=64273778, > > tx_1_to_64_packets=128, tx_256_to_511_packets=43670294, > > tx_512_to_1023_packets=153605, tx_65_to_127_packets=881272, > > tx_broadcast_packets=10, tx_bytes=25935295292, tx_dropped=0, > > tx_errors=0, tx_management_packets=0, tx_multicast_packets=153, > > tx_packets=109009906} > > > > > > sudo ovs-vsctl get Interface dpdk1 statistics > > {flow_director_filter_add_errors=126793, > > flow_director_filter_remove_errors=37969, mac_local_errors=0, > > mac_remote_errors=0, ovs_rx_qos_drops=0, ovs_tx_failure_drops=0, > > ovs_tx_invalid_hwol_drops=0, ovs_tx_mtu_exceeded_drops=0, > > ovs_tx_qos_drops=0, rx_128_to_255_packets=64435459, > > rx_1_to_64_packets=107843, rx_256_to_511_packets=230, > > rx_512_to_1023_packets=13, rx_65_to_127_packets=7049788, > > rx_broadcast_packets=199058, rx_bytes=12024342488, rx_crc_errors=0, > > rx_dropped=0, rx_errors=11, rx_fcoe_crc_errors=0, rx_fcoe_dropped=11, > > rx_fcoe_mbuf_allocation_errors=0, rx_fragment_errors=107843, > > rx_illegal_byte_errors=0, rx_jabber_errors=0, rx_length_errors=0, > > rx_mac_short_packet_dropped=1906, rx_management_dropped=0, > > rx_management_packets=13, rx_mbuf_allocation_errors=0, > > rx_missed_errors=0, rx_oversize_errors=0, rx_packets=71593333, > > rx_priority0_dropped=0, rx_priority0_mbuf_allocation_errors=1131, > > rx_priority1_dropped=0, rx_priority1_mbuf_allocation_errors=0, > > rx_priority2_dropped=0, rx_priority2_mbuf_allocation_errors=0, > > rx_priority3_dropped=0, rx_priority3_mbuf_allocation_errors=0, > > rx_priority4_dropped=0, rx_priority4_mbuf_allocation_errors=0, > > rx_priority5_dropped=0, rx_priority5_mbuf_allocation_errors=0, > > rx_priority6_dropped=0, rx_priority6_mbuf_allocation_errors=0, > > rx_priority7_dropped=0, rx_priority7_mbuf_allocation_errors=0, > > rx_undersize_errors=7049788, tx_128_to_255_packets=102664472, > > tx_1_to_64_packets=1906, tx_256_to_511_packets=68008814, > > tx_512_to_1023_packets=126793, tx_65_to_127_packets=1412435, > > tx_broadcast_packets=1464, tx_bytes=40693963125, tx_dropped=0, > > tx_errors=0, tx_management_packets=199058, tx_multicast_packets=146, > > tx_packets=172252389} > > The stats above don't show any significant drops that indicate ovs-dpdk > is the bottleneck, even with only using a single queue back from VM. > > There is a lot of rx undersize errors, so it could point to some issue > with your traffic gen or nic. > > > _______________________________________________ > > discuss mailing list > > [email protected] > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > _______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
