Hi Alec, I'm Pierrick Louin working in the team of François-Régis Menguy @orange-Labs.
In our last experiments, using the NFVbench application, we came across some issues and a workaround in trying to reach high performances from the bench at a high line rate. Maybe you can help us understanding what happens in the following observations. The raw traces are joined as text files. -------------------------------------------------------------------------------------------- Those reference tests are performed over a simple loopback link between the 2 ports of a NIC (either wired or through a hardware switch). We studied the two following cases: NIC Intel X520 (10 Gbits/s) NIC Mellanox ConnectX-5 (25 Gbits/s) Note that the list of the cpu threads reserved to the generator is not optimized. There is room for tuning... However this was not the point in these tests since, as we will see, the performance issue appears related to the RX process. The T-Rex generator will run with the same settings whether it is used standalone or wrapped in the NFVbench. TRex version: v2.59 NFVbench version: 3.5.0 (3.5.1.dev1) We have slightly patched the NFVbench code in order to make some processing now optional and/or configurable from the command line. --cores Override the T-Rex 'cores' parameter --service-mode Enable T-Rex service mode --extra-stats Enable extra flow stats (on high load traffic) --no-latency-stats Disable flow stats for latency traffic We consider the smallest packet size (64 bytes L2) in order to assess the maximum throughput achievable. Four tests are performed for each of the NICs tested - with a 100% and NDR rate. 1) Preliminary tests, performed with a basic scenario: 'pik.py' launched from a T-Rex console (derived from the script 'bench.py' coming with the T-Rex appli). 2) Tests performed using NFVbench (v3.5.0) in its native coding: High-rate generic streams (for BW measurement) and Low-rate streams (for Latency assessment) are configured into the T-Rex generator in order to allow stats computing on transmitted/received streams. 3) Tests performed using NFVbench where we have disabled into the 'trex_gen.py' script the instructions to tag the traffic generated for the purpose of further statistics (as far as we understand it) - this change is made in calls to STLStream(). 4) Tests performed using NFVbench where we keep the flow stats property for the latency streams only. FIRST ANALYSIS ************ The T-Rex test allows us to check that we have no bottleneck on the generator/analyzer + SUT side. The NFVbench results show acceptable performances only when dealing with the 10 Gbits/sec line. Using the NFVbench with its unmodified behaviour (case 2) the line rate is far from being reached with a 50 Gpbs line: 8.56 Gbits/s instead of 50 Gbits/s (L1) Unless there are some special reasons for activating a heavy flow stats RX processing, we suggest working in the case (4). We keep the latency assessment while the traffic counters seem to suffice at measuring the BW performances. Of course it may depend on the NIC capabilities for offloading traffic measurements. This is why we also make the flow stat activation optional. FURTHER ANALYSIS *************** However, looking closer to the performances obtained in the case (3), we can see a significantly reduced rate at the TX (and therefore RX) side: 19.09 Gbits/s instead of 20 Gbps (L1) - value is stable between launches 48.42 Gbits/s instead of 50 Gbps (L1) - actually variable between launches Note that the NDR measurement does not show any warning then. This is not our target case but it made me thinking... Thus, I tried a (5) case where we keep the flow stats for BW streams only. - the TX packet rate is reduced as in the case (3) for the Intel x520 - the throughput is limited by the line rate for the Mellanox ConnectX-5 <=> this is unexpected regarding to our hypotheses. CONCLUSION *********** It looks like we are missing something in our comprehension. Not sure that our workaround would not hide some side effects. So far, we can use it for our present needs. Waiting for reading from you. Thanks, Pierrick -------------------------------------------------------------------------------- [cid:image001.gif@01CF4791.BEC5FC70]<http://www.orange.com/> orange labs ORANGE/IMT/OLN/CISS/IIE/AMI Pierrick LOUIN 4, rue du Clos Courtel - BP 91226 35512 CESSON SEVIGNÉ cédex tél : +33 2 99 12 48 23 mob : +33 6 43 33 01 04 e-mail : pierrick.lo...@orange.com<mailto:pierrick.lo...@orange.com> -------------------------------------------------------------------------------- _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
########################################################################################## Looking for line rate performances using the traffic generator/analyser [NFVbench + T-Rex] ########################################################################################## =========================== NIC Intel X520 (10 Gbits/s) =========================== ****************************************************************************************** 1) T-Rex - alone ****************************************************************************************** T-Rex SERVER ------------ Configuration: - port_limit : 2 version : 2 zmq_pub_port : 4510 zmq_rpc_port : 4511 prefix : trex-local-R interfaces : ['0000:03:00.0','0000:03:00.1'] port_info: - dest_mac: 3c:fd:fe:cd:36:e1 src_mac: 3c:fd:fe:cd:36:e0 - dest_mac: 3c:fd:fe:cd:36:e0 src_mac: 3c:fd:fe:cd:36:e1 limit_memory: 2048 cores: 4 platform : master_thread_id : 0 latency_thread_id : 1 dual_if: - socket : 0 threads: [2,3,4,5,6,7,8,9,10,11,12,13,14,15] Starting T-Rex server to use 4 cores... (...) driver capability : TCP_UDP_OFFLOAD TSO set dpdk queues mode to DROP_QUE_FILTER Number of ports found: 2 (...) number of ports : 2 max cores for 2 ports : 4 tx queues per port : 6 (...) ------------------------------- RX core uses TX queue number 4 on all ports core, c-port, c-queue, s-port, s-queue, lat-queue ------------------------------------------ 1 0 0 1 0 5 2 0 1 1 1 255 3 0 2 1 2 255 4 0 3 1 3 255 ------------------------------- T-Rex CONSOLE ------------- Starting T-Rex console Configuration reported by the T-Rex console: Server Info: Server version: v2.59 @ STL Server mode: Stateless Server CPU: 4 x Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz Ports count: 2 x 10Gbps @ Ethernet 10G 2P X520 Adapter -=TRex Console v3.0=- trex>start -f stl/pik.py -m100% --port 0 1 -t size=64,vm=pik,vlan=1515 --force tui> Global Statistitcs connection : localhost, Port 4511 total_tx_L2 : 15.24 Gb/sec version : STL @ v2.59 total_tx_L1 : 20 Gb/sec cpu_util. : 98.81% @ 4 cores (4 per dual port) total_rx : 15.24 Gb/sec rx_cpu_util. : 0.0% / 0 pkt/sec total_pps : 29.77 Mpkt/sec async_util. : 0.05% / 1.49 KB/sec drop_rate : 0 b/sec total_cps. : 0 cps/sec queue_full : 870,938,247 pkts Port Statistics port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 10 Gb/s | 10 Gb/s | CPU util. | 98.81% | 98.81% | -- | | | Tx bps L2 | 7.62 Gbps | 7.62 Gbps | 15.24 Gbps Tx bps L1 | 10 Gbps | 10 Gbps | 20 Gbps Tx pps | 14.88 Mpps | 14.88 Mpps | 29.77 Mpps Line Util. | 100.02 % | 100.02 % | --- | | | Rx bps | 7.62 Gbps | 7.62 Gbps | 15.24 Gbps Rx pps | 14.88 Mpps | 14.88 Mpps | 29.77 Mpps ---- | | | opackets | 6161311235 | 6161344672 | 12322655907 ipackets | 4259022435 | 4259020502 | 8518042937 obytes | 394323919552 | 394326059392 | 788649978944 ibytes | 272577436288 | 272577312576 | 545154748864 tx-pkts | 6.16 Gpkts | 6.16 Gpkts | 12.32 Gpkts rx-pkts | 4.26 Gpkts | 4.26 Gpkts | 8.52 Gpkts tx-bytes | 394.32 GB | 394.33 GB | 788.65 GB rx-bytes | 272.58 GB | 272.58 GB | 545.15 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0 status: - ****************************************************************************************** 2) NFVbench - all flow-stats ****************************************************************************************** =============== 2.1 rate = 100% =============== nfvbench -c RRR.cfg --rate=100% --duration=30 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=4 --extra-stats ========== NFVBench Summary ========== Date: 2019-07-29 21:06:26 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-R Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----------------+-------------+----------------------+----------------------+----------------------+ | L2 Frame Size | Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=================+=============+======================+======================+======================+ | 64 | 0.0000% | 2,609 | 10 | 4,419 | +-----------------+-------------+----------------------+----------------------+----------------------+ L2 frame size: 64 Run Config: +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Direction | Requested TX Rate (bps) | Actual TX Rate (bps) | RX Rate (bps) | Requested TX Rate (pps) | Actual TX Rate (pps) | RX Rate (pps) | +=============+===========================+========================+=================+===========================+========================+=================+ | Forward | 10.0000 Gbps | 10.0000 Gbps | 10.0000 Gbps | 14,880,952 pps | 14,880,952 pps | 14,880,952 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Reverse | 10.0000 Gbps | 10.0000 Gbps | 10.0000 Gbps | 14,880,952 pps | 14,880,952 pps | 14,880,952 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Total | 20.0000 Gbps | 20.0000 Gbps | 20.0000 Gbps | 29,761,904 pps | 29,761,904 pps | 29,761,904 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ Forward Chain Packet Counters and Latency: +---------+--------------+--------------------------+------------+------------+------------+ | Chain | TRex.TX.p0 | TRex.RX.p1 | Avg lat. | Min lat. | Max lat. | +=========+==============+==========================+============+============+============+ | 0 | 446,428,563 | -339,583,843 (-76.0668%) | 2,606 usec | 10 usec | 4,419 usec | +---------+--------------+--------------------------+------------+------------+------------+ Reverse Chain Packet Counters and Latency: +---------+--------------+--------------------------+------------+------------+------------+ | Chain | TRex.TX.p1 | TRex.RX.p0 | Avg lat. | Min lat. | Max lat. | +=========+==============+==========================+============+============+============+ | 0 | 446,428,563 | -339,587,392 (-76.0676%) | 2,613 usec | 20 usec | 4,060 usec | +---------+--------------+--------------------------+------------+------------+------------+ =============== 2.2 rate = NDR =============== nfvbench -c RRR.cfg --rate=NDR --duration=10 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=4 --extra-stats ========== NFVBench Summary ========== Date: 2019-07-29 21:47:22 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-R Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Measurement Parameters: NDR: 0.001 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ | - | L2 Frame Size | Rate (fwd+rev) | Rate (fwd+rev) | Avg Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=====+=================+==================+==================+=================+======================+======================+======================+ | NDR | 64 | 20.0000 Gbps | 29,761,904 pps | 0.0000% | 2,597 | 70 | 4,767 | +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ L2 frame size: 64 NDR search duration: 12 seconds ****************************************************************************************** 3) NFVbench - no flow-stats ****************************************************************************************** =============== 3.1 rate = 100% =============== nfvbench -c RRR.cfg --rate=100% --duration=30 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=4 --no-latency-stats ========== NFVBench Summary ========== Date: 2019-07-29 21:09:17 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-R Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----------------+-------------+----------------------+----------------------+----------------------+ | L2 Frame Size | Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=================+=============+======================+======================+======================+ | 64 | 0.0000% | 0 | 0 | 0 | +-----------------+-------------+----------------------+----------------------+----------------------+ L2 frame size: 64 Run Config: +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Direction | Requested TX Rate (bps) | Actual TX Rate (bps) | RX Rate (bps) | Requested TX Rate (pps) | Actual TX Rate (pps) | RX Rate (pps) | +=============+===========================+========================+=================+===========================+========================+=================+ | Forward | 10.0000 Gbps | 9.5466 Gbps | 9.5466 Gbps | 14,880,952 pps | 14,206,299 pps | 14,206,299 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Reverse | 10.0000 Gbps | 9.5465 Gbps | 9.5465 Gbps | 14,880,952 pps | 14,206,125 pps | 14,206,125 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Total | 20.0000 Gbps | 19.0931 Gbps | 19.0931 Gbps | 29,761,904 pps | 28,412,424 pps | 28,412,424 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ Forward Chain Packet Counters and Latency: +---------+--------------+-------------------------+ | Chain | TRex.TX.p0 | TRex.RX.p1 | +=========+==============+=========================+ | 0 | 0 | +426,188,980 (+0.0000%) | +---------+--------------+-------------------------+ Reverse Chain Packet Counters and Latency: +---------+--------------+-------------------------+ | Chain | TRex.TX.p1 | TRex.RX.p0 | +=========+==============+=========================+ | 0 | 0 | +426,183,751 (+0.0000%) | +---------+--------------+-------------------------+ =============== 3.2 rate = NDR =============== nfvbench -c RRR.cfg --rate=NDR --duration=10 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=4 --no-latency-stats ========== NFVBench Summary ========== Date: 2019-07-29 21:48:40 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-R Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Measurement Parameters: NDR: 0.001 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ | - | L2 Frame Size | Rate (fwd+rev) | Rate (fwd+rev) | Avg Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=====+=================+==================+==================+=================+======================+======================+======================+ | NDR | 64 | 20.0000 Gbps | 29,761,904 pps | 0.0000% | 0 | 0 | 0 | +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ L2 frame size: 64 NDR search duration: 12 seconds ****************************************************************************************** 4) NFVbench - latency flow-stats only ****************************************************************************************** =============== 4.1 rate = 100% =============== nfvbench -c RRR.cfg --rate=100% --duration=30 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=4 ========== NFVBench Summary ========== Date: 2019-07-29 21:11:05 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-R Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----------------+-------------+----------------------+----------------------+----------------------+ | L2 Frame Size | Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=================+=============+======================+======================+======================+ | 64 | 0.0000% | 640 | 2 | 1,358 | +-----------------+-------------+----------------------+----------------------+----------------------+ L2 frame size: 64 Run Config: +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Direction | Requested TX Rate (bps) | Actual TX Rate (bps) | RX Rate (bps) | Requested TX Rate (pps) | Actual TX Rate (pps) | RX Rate (pps) | +=============+===========================+========================+=================+===========================+========================+=================+ | Forward | 10.0000 Gbps | 10.0000 Gbps | 10.0000 Gbps | 14,880,952 pps | 14,880,950 pps | 14,880,950 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Reverse | 10.0000 Gbps | 10.0000 Gbps | 10.0000 Gbps | 14,880,952 pps | 14,880,950 pps | 14,880,950 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Total | 20.0000 Gbps | 20.0000 Gbps | 20.0000 Gbps | 29,761,904 pps | 29,761,900 pps | 29,761,900 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ Forward Chain Packet Counters and Latency: +---------+--------------+---------------------+------------+------------+------------+ | Chain | TRex.TX.p0 | TRex.RX.p1 | Avg lat. | Min lat. | Max lat. | +=========+==============+=====================+============+============+============+ | 0 | 30,001 | -11,082 (-36.9388%) | 640 usec | 2 usec | 1,358 usec | +---------+--------------+---------------------+------------+------------+------------+ Reverse Chain Packet Counters and Latency: +---------+--------------+---------------------+------------+------------+------------+ | Chain | TRex.TX.p1 | TRex.RX.p0 | Avg lat. | Min lat. | Max lat. | +=========+==============+=====================+============+============+============+ | 0 | 30,001 | -10,733 (-35.7755%) | 640 usec | 2 usec | 1,259 usec | +---------+--------------+---------------------+------------+------------+------------+ =============== 4.2 rate = NDR =============== nfvbench -c RRR.cfg --rate=NDR --duration=10 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=4 ========== NFVBench Summary ========== Date: 2019-07-29 21:49:37 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-R Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Measurement Parameters: NDR: 0.001 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ | - | L2 Frame Size | Rate (fwd+rev) | Rate (fwd+rev) | Avg Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=====+=================+==================+==================+=================+======================+======================+======================+ | NDR | 64 | 20.0000 Gbps | 29,761,904 pps | 0.0000% | 639 | 2 | 1,344 | +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ L2 frame size: 64 NDR search duration: 12 seconds ****************************************************************************************** 5) NFVbench - extra flow-stats only ****************************************************************************************** =============== 5.1 rate = 100% =============== nfvbench -c RRR.cfg --rate=100% --duration=30 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=4 --no-latency-stats --extra-stats ========== NFVBench Summary ========== Date: 2019-07-30 14:11:42 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-R Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----------------+-------------+----------------------+----------------------+----------------------+ | L2 Frame Size | Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=================+=============+======================+======================+======================+ | 64 | 0.0000% | 0 | 0 | 0 | +-----------------+-------------+----------------------+----------------------+----------------------+ L2 frame size: 64 Run Config: +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Direction | Requested TX Rate (bps) | Actual TX Rate (bps) | RX Rate (bps) | Requested TX Rate (pps) | Actual TX Rate (pps) | RX Rate (pps) | +=============+===========================+========================+=================+===========================+========================+=================+ | Forward | 10.0000 Gbps | 9.5467 Gbps | 9.5467 Gbps | 14,880,952 pps | 14,206,403 pps | 14,206,403 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Reverse | 10.0000 Gbps | 9.5465 Gbps | 9.5465 Gbps | 14,880,952 pps | 14,206,098 pps | 14,206,098 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Total | 20.0000 Gbps | 19.0932 Gbps | 19.0932 Gbps | 29,761,904 pps | 28,412,501 pps | 28,412,501 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ Forward Chain Packet Counters and Latency: +---------+--------------+----------------------+ | Chain | TRex.TX.p0 | TRex.RX.p1 | +=========+==============+======================+ | 0 | 425,767 | -161,929 (-38.0323%) | +---------+--------------+----------------------+ Reverse Chain Packet Counters and Latency: +---------+--------------+----------------------+ | Chain | TRex.TX.p1 | TRex.RX.p0 | +=========+==============+======================+ | 0 | 425,758 | -158,370 (-37.1972%) | +---------+--------------+----------------------+ =============== 5.2 rate = NDR =============== nfvbench -c RRR.cfg --rate=NDR --duration=10 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=4 --no-latency-stats --extra-stats ========== NFVBench Summary ========== Date: 2019-07-30 14:09:09 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-R Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Measurement Parameters: NDR: 0.001 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ | - | L2 Frame Size | Rate (fwd+rev) | Rate (fwd+rev) | Avg Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=====+=================+==================+==================+=================+======================+======================+======================+ | NDR | 64 | 20.0000 Gbps | 29,761,904 pps | 0.0000% | 0 | 0 | 0 | +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ L2 frame size: 64 NDR search duration: 12 seconds
########################################################################################## Looking for line rate performances using the traffic generator/analyser [NFVbench + T-Rex] ########################################################################################## ==================================== NIC Mellanox ConnectX-5 (25 Gbits/s) ==================================== ****************************************************************************************** 1) T-Rex - alone ****************************************************************************************** T-Rex SERVER ------------ Configuration: - port_limit : 2 version : 2 zmq_pub_port : 4500 zmq_rpc_port : 4501 prefix : trex-local-L interfaces : ['0000:04:00.0','0000:04:00.1'] port_info: - dest_mac: 98:03:9b:c9:bc:c5 src_mac: 98:03:9b:c9:bc:c4 - dest_mac: 98:03:9b:c9:bc:c4 src_mac: 98:03:9b:c9:bc:c5 limit_memory: 2048 cores: 6 platform : master_thread_id : 0 latency_thread_id : 1 dual_if: - socket : 0 threads: [2,3,4,5,6,7,8,9,10,11,12,13,14,15] Starting T-Rex server to use 6 cores... (...) driver capability : TCP_UDP_OFFLOAD TSO set dpdk queues mode to DROP_QUE_FILTER Number of ports found: 2 (...) number of ports : 2 max cores for 2 ports : 6 tx queues per port : 8 (...) ------------------------------- RX core uses TX queue number 6 on all ports core, c-port, c-queue, s-port, s-queue, lat-queue ------------------------------------------ 1 0 0 1 0 7 2 0 1 1 1 255 3 0 2 1 2 255 4 0 3 1 3 255 5 0 4 1 4 255 6 0 5 1 5 255 T-Rex CONSOLE ------------- Starting T-Rex console Configuration reported by the T-Rex console: Server Info: Server version: v2.59 @ STL Server mode: Stateless Server CPU: 6 x Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz Ports count: 2 x 10Gbps @ MT27800 Family [ConnectX-5] ^ note: the default speed is not 25 Gbps -=TRex Console v3.0=- trex>start -f stl/pik.py -m100% --port 0 1 -t size=64,vm=pik --force tui> Global Statistitcs connection : localhost, Port 4501 total_tx_L2 : 38.09 Gb/sec version : STL @ v2.59 total_tx_L1 : 49.99 Gb/sec cpu_util. : 63.25% @ 6 cores (6 per dual port) total_rx : 38.09 Gb/sec rx_cpu_util. : 0.0% / 0 pkt/sec total_pps : 74.39 Mpkt/sec async_util. : 0.04% / 1.47 KB/sec drop_rate : 0 b/sec total_cps. : 0 cps/sec queue_full : 26,365 pkts Port Statistics port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 25 Gb/s | 25 Gb/s | CPU util. | 63.25% | 63.25% | -- | | | Tx bps L2 | 19.04 Gbps | 19.04 Gbps | 38.09 Gbps Tx bps L1 | 25 Gbps | 25 Gbps | 49.99 Gbps Tx pps | 37.2 Mpps | 37.2 Mpps | 74.39 Mpps Line Util. | 99.98 % | 99.98 % | --- | | | Rx bps | 19.04 Gbps | 19.04 Gbps | 38.09 Gbps Rx pps | 37.2 Mpps | 37.2 Mpps | 74.39 Mpps ---- | | | opackets | 34068259994 | 34068282338 | 68136542332 ipackets | 34068274921 | 34068267412 | 68136542333 obytes | 2180368639616 | 2180370069692 | 4360738709308 ibytes | 2180369594944 | 2180369114368 | 4360738709312 tx-pkts | 34.07 Gpkts | 34.07 Gpkts | 68.14 Gpkts rx-pkts | 34.07 Gpkts | 34.07 Gpkts | 68.14 Gpkts tx-bytes | 2.18 TB | 2.18 TB | 4.36 TB rx-bytes | 2.18 TB | 2.18 TB | 4.36 TB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0 status: \ ****************************************************************************************** 2) NFVbench - all flow-stats ****************************************************************************************** =============== 2.1 rate = 100% =============== nfvbench -c LLL.cfg --rate=100% --duration=30 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=6 --extra-stats ========== NFVBench Summary ========== Date: 2019-07-29 21:15:30 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-L Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----------------+-------------+----------------------+----------------------+----------------------+ | L2 Frame Size | Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=================+=============+======================+======================+======================+ | 64 | 82.8872% | 381 | 60 | 775 | +-----------------+-------------+----------------------+----------------------+----------------------+ L2 frame size: 64 Run Config: +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Direction | Requested TX Rate (bps) | Actual TX Rate (bps) | RX Rate (bps) | Requested TX Rate (pps) | Actual TX Rate (pps) | RX Rate (pps) | +=============+===========================+========================+=================+===========================+========================+=================+ | Forward | 25.0000 Gbps | 25.0000 Gbps | 4.2784 Gbps | 37,202,380 pps | 37,202,379 pps | 6,366,711 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Reverse | 25.0000 Gbps | 25.0000 Gbps | 4.2780 Gbps | 37,202,380 pps | 37,202,339 pps | 6,366,048 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Total | 50.0000 Gbps | 50.0000 Gbps | 8.5564 Gbps | 74,404,760 pps | 74,404,718 pps | 12,732,759 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ Forward Chain Packet Counters and Latency: +---------+---------------+----------------------------+------------+------------+------------+ | Chain | TRex.TX.p0 | TRex.RX.p1 | Avg lat. | Min lat. | Max lat. | +=========+===============+============================+============+============+============+ | 0 | 1,116,071,397 | -1,016,292,896 (-91.0598%) | 372 usec | 100 usec | 775 usec | +---------+---------------+----------------------------+------------+------------+------------+ Reverse Chain Packet Counters and Latency: +---------+---------------+----------------------------+------------+------------+------------+ | Chain | TRex.TX.p1 | TRex.RX.p0 | Avg lat. | Min lat. | Max lat. | +=========+===============+============================+============+============+============+ | 0 | 1,116,070,178 | -1,016,283,793 (-91.0591%) | 393 usec | 60 usec | 760 usec | +---------+---------------+----------------------------+------------+------------+------------+ =============== 2.2 rate = NDR =============== nfvbench -c LLL.cfg --rate=NDR --duration=10 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=6 --extra-stats ========== NFVBench Summary ========== Date: 2019-07-29 21:52:25 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-L Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Measurement Parameters: NDR: 0.001 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ | - | L2 Frame Size | Rate (fwd+rev) | Rate (fwd+rev) | Avg Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=====+=================+==================+==================+=================+======================+======================+======================+ | NDR | 64 | 10.2051 Gbps | 15,186,128 pps | 0.0000% | 158 | 2 | 377 | +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ L2 frame size: 64 NDR search duration: 133 seconds ****************************************************************************************** 3) NFVbench - no flow-stats ****************************************************************************************** =============== 3.1 rate = 100% =============== nfvbench -c LLL.cfg --rate=100% --duration=30 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=6 --no-latency-stats ========== NFVBench Summary ========== Date: 2019-07-30 15:07:26 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-L Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----------------+-------------+----------------------+----------------------+----------------------+ | L2 Frame Size | Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=================+=============+======================+======================+======================+ | 64 | 0.0000% | 0 | 0 | 0 | +-----------------+-------------+----------------------+----------------------+----------------------+ L2 frame size: 64 Run Config: +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Direction | Requested TX Rate (bps) | Actual TX Rate (bps) | RX Rate (bps) | Requested TX Rate (pps) | Actual TX Rate (pps) | RX Rate (pps) | +=============+===========================+========================+=================+===========================+========================+=================+ | Forward | 25.0000 Gbps | 24.2096 Gbps | 24.2096 Gbps | 37,202,380 pps | 36,026,116 pps | 36,026,116 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Reverse | 25.0000 Gbps | 24.2091 Gbps | 24.2091 Gbps | 37,202,380 pps | 36,025,396 pps | 36,025,396 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Total | 50.0000 Gbps | 48.4186 Gbps | 48.4186 Gbps | 74,404,760 pps | 72,051,512 pps | 72,051,512 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ Forward Chain Packet Counters and Latency: +---------+--------------+---------------------------+ | Chain | TRex.TX.p0 | TRex.RX.p1 | +=========+==============+===========================+ | 0 | 0 | +1,080,783,505 (+0.0000%) | +---------+--------------+---------------------------+ Reverse Chain Packet Counters and Latency: +---------+--------------+---------------------------+ | Chain | TRex.TX.p1 | TRex.RX.p0 | +=========+==============+===========================+ | 0 | 0 | +1,080,761,902 (+0.0000%) | +---------+--------------+---------------------------+ =============== 3.2 rate = NDR =============== nfvbench -c LLL.cfg --rate=NDR --duration=10 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=6 --no-latency-stats ========== NFVBench Summary ========== Date: 2019-07-30 15:10:19 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-L Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Measurement Parameters: NDR: 0.001 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ | - | L2 Frame Size | Rate (fwd+rev) | Rate (fwd+rev) | Avg Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=====+=================+==================+==================+=================+======================+======================+======================+ | NDR | 64 | 50.0000 Gbps | 74,404,760 pps | 0.0000% | 0 | 0 | 0 | +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ L2 frame size: 64 NDR search duration: 12 seconds [****************************************************************************************** 4) NFVbench - latency flow-stats only ****************************************************************************************** =============== 4.1 rate = 100% =============== nfvbench -c LLL.cfg --rate=100% --duration=30 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=6 ========== NFVBench Summary ========== Date: 2019-07-30 14:55:19 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-L Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----------------+-------------+----------------------+----------------------+----------------------+ | L2 Frame Size | Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=================+=============+======================+======================+======================+ | 64 | 0.0000% | 3 | 2 | 604 | +-----------------+-------------+----------------------+----------------------+----------------------+ L2 frame size: 64 Run Config: +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Direction | Requested TX Rate (bps) | Actual TX Rate (bps) | RX Rate (bps) | Requested TX Rate (pps) | Actual TX Rate (pps) | RX Rate (pps) | +=============+===========================+========================+=================+===========================+========================+=================+ | Forward | 25.0000 Gbps | 25.0000 Gbps | 25.0000 Gbps | 37,202,380 pps | 37,202,381 pps | 37,202,381 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Reverse | 25.0000 Gbps | 25.0000 Gbps | 25.0000 Gbps | 37,202,380 pps | 37,202,381 pps | 37,202,381 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Total | 50.0000 Gbps | 50.0000 Gbps | 50.0000 Gbps | 74,404,760 pps | 74,404,762 pps | 74,404,762 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ Forward Chain Packet Counters and Latency: +---------+--------------+--------------+------------+------------+------------+ | Chain | TRex.TX.p0 | TRex.RX.p1 | Avg lat. | Min lat. | Max lat. | +=========+==============+==============+============+============+============+ | 0 | 30,001 | 30,001 | 3 usec | 2 usec | 602 usec | +---------+--------------+--------------+------------+------------+------------+ Reverse Chain Packet Counters and Latency: +---------+--------------+--------------+------------+------------+------------+ | Chain | TRex.TX.p1 | TRex.RX.p0 | Avg lat. | Min lat. | Max lat. | +=========+==============+==============+============+============+============+ | 0 | 30,001 | 30,001 | 3 usec | 2 usec | 604 usec | +---------+--------------+--------------+------------+------------+------------+ =============== 4.2 rate = NDR =============== nfvbench -c LLL.cfg --rate=NDR --duration=10 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=6 ========== NFVBench Summary ========== Date: 2019-07-30 14:54:25 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-L Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Measurement Parameters: NDR: 0.001 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ | - | L2 Frame Size | Rate (fwd+rev) | Rate (fwd+rev) | Avg Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=====+=================+==================+==================+=================+======================+======================+======================+ | NDR | 64 | 50.0000 Gbps | 74,404,760 pps | 0.0000% | 2 | 2 | 7,970 | +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ L2 frame size: 64 NDR search duration: 12 seconds ****************************************************************************************** 5) NFVbench - extra flow-stats only ****************************************************************************************** =============== 5.1 rate = 100% =============== nfvbench -c LLL.cfg --rate=NDR --duration=10 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=6 --no-latency-stats --extra-stats ========== NFVBench Summary ========== Date: 2019-07-30 14:46:32 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-L Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----------------+-------------+----------------------+----------------------+----------------------+ | L2 Frame Size | Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=================+=============+======================+======================+======================+ | 64 | 0.0000% | 0 | 0 | 0 | +-----------------+-------------+----------------------+----------------------+----------------------+ L2 frame size: 64 Run Config: +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Direction | Requested TX Rate (bps) | Actual TX Rate (bps) | RX Rate (bps) | Requested TX Rate (pps) | Actual TX Rate (pps) | RX Rate (pps) | +=============+===========================+========================+=================+===========================+========================+=================+ | Forward | 25.0000 Gbps | 25.0000 Gbps | 25.0000 Gbps | 37,202,380 pps | 37,202,379 pps | 37,202,379 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Reverse | 25.0000 Gbps | 25.0000 Gbps | 25.0000 Gbps | 37,202,380 pps | 37,202,379 pps | 37,202,379 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ | Total | 50.0000 Gbps | 50.0000 Gbps | 50.0000 Gbps | 74,404,760 pps | 74,404,758 pps | 74,404,758 pps | +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+ Forward Chain Packet Counters and Latency: +---------+--------------+--------------+ | Chain | TRex.TX.p0 | TRex.RX.p1 | +=========+==============+==============+ | 0 | 1,114,957 | 1,114,957 | +---------+--------------+--------------+ Reverse Chain Packet Counters and Latency: +---------+--------------+--------------+ | Chain | TRex.TX.p1 | TRex.RX.p0 | +=========+==============+==============+ | 0 | 1,114,957 | 1,114,957 | +---------+--------------+--------------+ =============== 5.2 rate = NDR =============== nfvbench -c LLL.cfg --rate=NDR --duration=10 --interval=1 --flow-count=10000 -fs=64 -scc=1 --cores=6 --no-latency-stats --extra-stats ========== NFVBench Summary ========== Date: 2019-07-30 14:47:33 NFVBench version 3.5.1.dev1 Openstack Neutron: vSwitch: Encapsulation: Benchmarks: > Networks: > Components: > Traffic Generator: Profile: trex-local-L Tool: TRex > Versions: > Traffic_Generator: build_date: Jul 9 2019 version: v2.59 built_by: hhaim mode: STL build_time: 13:03:30 > Measurement Parameters: NDR: 0.001 > Service chain: > EXT: > Traffic: Profile: custom_traffic_profile Bidirectional: True Flow count: 10000 Service chains count: 1 Compute nodes: [] Run Summary: +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ | - | L2 Frame Size | Rate (fwd+rev) | Rate (fwd+rev) | Avg Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) | +=====+=================+==================+==================+=================+======================+======================+======================+ | NDR | 64 | 50.0000 Gbps | 74,404,760 pps | 0.0000% | 0 | 0 | 0 | +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+ L2 frame size: 64 NDR search duration: 12 seconds
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#23421): https://lists.opnfv.org/g/opnfv-tech-discuss/message/23421 Mute This Topic: https://lists.opnfv.org/mt/32684476/21656 Mute #nfvbench: https://lists.opnfv.org/mk?hashtag=nfvbench&subid=2783016 Group Owner: opnfv-tech-discuss+ow...@lists.opnfv.org Unsubscribe: https://lists.opnfv.org/g/opnfv-tech-discuss/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-