On 30/10/2021 06:07, Satish Patel wrote:
Folks,

I have configured ovs-dpdk to replace sriov deployment for bonding
support. everything good but somehow as soon as i start hitting
200kpps rate i start seeing packet drop.

I have configured CPU isolation as per documentation to assign a
dedicated pmd thread. I have assigned 8 dedicated PMD threads but
still performance is very poor.

I created an 8vCPU vm on openstack using dpdk and running some
workload using an in-house application, during the 200kpps packet rate
I noticed my all PMD cpu showing high CPU processing cycles.


Not sure what you class as 'high CPU processing cycles', but if you mean at the point of hitting a CPU bottleneck and the stats below are taken with contrant traffic, none of them would indicate that is the case.

  $ ovs-vswitchd -V
ovs-vswitchd (Open vSwitch) 2.13.3
DPDK 19.11.7

In the following output what does these queue-id:0 to 8 and why only
the first queue is in use but not others, they are always zero. What
does this mean?


It means the application is only sending packets to OVS on a single queue. It may be that the application does not do multi-queue, or because the application sends on the same queue number that it receives traffic on and it is only receiving traffic on one queue.

ovs-appctl dpif-netdev/pmd-rxq-show
pmd thread numa_id 0 core_id 2:
   isolated : false
   port: vhu1c3bf17a-01    queue-id:  0 (enabled)   pmd usage:  0 %
   port: vhu1c3bf17a-01    queue-id:  1 (enabled)   pmd usage:  0 %
   port: vhu6b7daba9-1a    queue-id:  2 (disabled)  pmd usage:  0 %
   port: vhu6b7daba9-1a    queue-id:  3 (disabled)  pmd usage:  0 %
pmd thread numa_id 1 core_id 3:
   isolated : false
pmd thread numa_id 0 core_id 22:
   isolated : false
   port: vhu1c3bf17a-01    queue-id:  3 (enabled)   pmd usage:  0 %
   port: vhu1c3bf17a-01    queue-id:  6 (enabled)   pmd usage:  0 %
   port: vhu6b7daba9-1a    queue-id:  0 (enabled)   pmd usage: 54 %
   port: vhu6b7daba9-1a    queue-id:  5 (disabled)  pmd usage:  0 %
pmd thread numa_id 1 core_id 23:
   isolated : false
   port: dpdk1             queue-id:  0 (enabled)   pmd usage:  3 %
pmd thread numa_id 0 core_id 26:
   isolated : false
   port: vhu1c3bf17a-01    queue-id:  2 (enabled)   pmd usage:  0 %
   port: vhu1c3bf17a-01    queue-id:  7 (enabled)   pmd usage:  0 %
   port: vhu6b7daba9-1a    queue-id:  1 (disabled)  pmd usage:  0 %
   port: vhu6b7daba9-1a    queue-id:  4 (disabled)  pmd usage:  0 %
pmd thread numa_id 1 core_id 27:
   isolated : false
pmd thread numa_id 0 core_id 46:
   isolated : false
   port: dpdk0             queue-id:  0 (enabled)   pmd usage:  27 %
   port: vhu1c3bf17a-01    queue-id:  4 (enabled)   pmd usage:  0 %
   port: vhu1c3bf17a-01    queue-id:  5 (enabled)   pmd usage:  0 %
   port: vhu6b7daba9-1a    queue-id:  6 (disabled)  pmd usage:  0 %
   port: vhu6b7daba9-1a    queue-id:  7 (disabled)  pmd usage:  0 %
pmd thread numa_id 1 core_id 47:
   isolated : false


$ ovs-appctl dpif-netdev/pmd-stats-clear && sleep 10 && ovs-appctl
dpif-netdev/pmd-stats-show | grep "processing cycles:"
   processing cycles: 1697952 (0.01%)
   processing cycles: 12726856558 (74.96%)
   processing cycles: 4259431602 (19.40%)
   processing cycles: 512666 (0.00%)
   processing cycles: 6324848608 (37.81%)

Does processing cycles mean my PMD is under stress? but i am only
hitting 200kpps rate?


This is my dpdk0 and dpdk1 port statistics

sudo ovs-vsctl get Interface dpdk0 statistics
{flow_director_filter_add_errors=153605,
flow_director_filter_remove_errors=30829, mac_local_errors=0,
mac_remote_errors=0, ovs_rx_qos_drops=0, ovs_tx_failure_drops=0,
ovs_tx_invalid_hwol_drops=0, ovs_tx_mtu_exceeded_drops=0,
ovs_tx_qos_drops=0, rx_128_to_255_packets=64338613,
rx_1_to_64_packets=367, rx_256_to_511_packets=116298,
rx_512_to_1023_packets=31264, rx_65_to_127_packets=6990079,
rx_broadcast_packets=0, rx_bytes=12124930385, rx_crc_errors=0,
rx_dropped=0, rx_errors=12, rx_fcoe_crc_errors=0, rx_fcoe_dropped=12,
rx_fcoe_mbuf_allocation_errors=0, rx_fragment_errors=367,
rx_illegal_byte_errors=0, rx_jabber_errors=0, rx_length_errors=0,
rx_mac_short_packet_dropped=128, rx_management_dropped=35741,
rx_management_packets=31264, rx_mbuf_allocation_errors=0,
rx_missed_errors=0, rx_oversize_errors=0, rx_packets=71512362,
rx_priority0_dropped=0, rx_priority0_mbuf_allocation_errors=1096,
rx_priority1_dropped=0, rx_priority1_mbuf_allocation_errors=0,
rx_priority2_dropped=0, rx_priority2_mbuf_allocation_errors=0,
rx_priority3_dropped=0, rx_priority3_mbuf_allocation_errors=0,
rx_priority4_dropped=0, rx_priority4_mbuf_allocation_errors=0,
rx_priority5_dropped=0, rx_priority5_mbuf_allocation_errors=0,
rx_priority6_dropped=0, rx_priority6_mbuf_allocation_errors=0,
rx_priority7_dropped=0, rx_priority7_mbuf_allocation_errors=0,
rx_undersize_errors=6990079, tx_128_to_255_packets=64273778,
tx_1_to_64_packets=128, tx_256_to_511_packets=43670294,
tx_512_to_1023_packets=153605, tx_65_to_127_packets=881272,
tx_broadcast_packets=10, tx_bytes=25935295292, tx_dropped=0,
tx_errors=0, tx_management_packets=0, tx_multicast_packets=153,
tx_packets=109009906}


sudo ovs-vsctl get Interface dpdk1 statistics
{flow_director_filter_add_errors=126793,
flow_director_filter_remove_errors=37969, mac_local_errors=0,
mac_remote_errors=0, ovs_rx_qos_drops=0, ovs_tx_failure_drops=0,
ovs_tx_invalid_hwol_drops=0, ovs_tx_mtu_exceeded_drops=0,
ovs_tx_qos_drops=0, rx_128_to_255_packets=64435459,
rx_1_to_64_packets=107843, rx_256_to_511_packets=230,
rx_512_to_1023_packets=13, rx_65_to_127_packets=7049788,
rx_broadcast_packets=199058, rx_bytes=12024342488, rx_crc_errors=0,
rx_dropped=0, rx_errors=11, rx_fcoe_crc_errors=0, rx_fcoe_dropped=11,
rx_fcoe_mbuf_allocation_errors=0, rx_fragment_errors=107843,
rx_illegal_byte_errors=0, rx_jabber_errors=0, rx_length_errors=0,
rx_mac_short_packet_dropped=1906, rx_management_dropped=0,
rx_management_packets=13, rx_mbuf_allocation_errors=0,
rx_missed_errors=0, rx_oversize_errors=0, rx_packets=71593333,
rx_priority0_dropped=0, rx_priority0_mbuf_allocation_errors=1131,
rx_priority1_dropped=0, rx_priority1_mbuf_allocation_errors=0,
rx_priority2_dropped=0, rx_priority2_mbuf_allocation_errors=0,
rx_priority3_dropped=0, rx_priority3_mbuf_allocation_errors=0,
rx_priority4_dropped=0, rx_priority4_mbuf_allocation_errors=0,
rx_priority5_dropped=0, rx_priority5_mbuf_allocation_errors=0,
rx_priority6_dropped=0, rx_priority6_mbuf_allocation_errors=0,
rx_priority7_dropped=0, rx_priority7_mbuf_allocation_errors=0,
rx_undersize_errors=7049788, tx_128_to_255_packets=102664472,
tx_1_to_64_packets=1906, tx_256_to_511_packets=68008814,
tx_512_to_1023_packets=126793, tx_65_to_127_packets=1412435,
tx_broadcast_packets=1464, tx_bytes=40693963125, tx_dropped=0,
tx_errors=0, tx_management_packets=199058, tx_multicast_packets=146,
tx_packets=172252389}

The stats above don't show any significant drops that indicate ovs-dpdk is the bottleneck, even with only using a single queue back from VM.

There is a lot of rx undersize errors, so it could point to some issue with your traffic gen or nic.

_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss


_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Reply via email to