Hi All, FD.io CSIT-1908.37 report has been published on FD.io docs site:
html: https://docs.fd.io/csit/rls1908/report/ pdf: https://docs.fd.io/csit/rls1908/report/_static/archive/csit_rls1908.pdf Tibor -----Original Message----- From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Maciek Konstantynowicz (mkonstan) via Lists.Fd.Io Sent: Wednesday, September 4, 2019 10:17 PM To: csit-dev <csit-...@lists.fd.io>; vpp-dev <vpp-dev@lists.fd.io>; honeycomb-dev <honeycomb-...@lists.fd.io>; hc2...@lists.fd.io; govpp-...@lists.fd.io Cc: vpp-dev@lists.fd.io Subject: [vpp-dev] Published: FD.io CSIT-1908 Release Report Hi All, FD.io CSIT-1908 report has been published on FD.io docs site: html: https://docs.fd.io/csit/rls1908/report/ pdf: https://docs.fd.io/csit/rls1908/report/_static/archive/csit_rls1908.pdf Many thanks to All in CSIT, VPP and wider FD.io community who contributed and worked hard to make CSIT-1908 happen! Below two summaries: - CSIT-1908 Release Summary, a high-level summary. - Points of Note in CSIT-1908 Report, with specific links to report. Welcome all comments, best by email to csit-...@lists.fd.io. Cheers, -Maciek ------------------------------------------------------------------------ CSIT-1908 Release Summary ------------------------------------------------------------------------ 1. CSIT-1908 Report - html link: https://docs.fd.io/csit/rls1908/report/ - pdf link: https://docs.fd.io/csit/rls1908/report/_static/archive/csit_rls1908.pdf 2. New Tests - VM/VNF service chains with DPDK Testpmd and VPP L2/IPv4 workloads and external VXLAN encapsulation. - IPsec with new VPP native cipher algorithms, baseline and large scale (up to 60k tunnels). - VM/VNF service chains with VPP IPsec workloads, baseline and horizontal scaling (experimental, in-testing). - GBP (Group Based Policy) with external dot1q encapsulation. - Extended test coverage with VPP native AVF driver: IPv4 scale tests. - Refreshed VPP TCP/HTTP tests. - Number of VPP functional device tests running in container based environment. - Good VPP PAPI (Python API) test coverage, PAPI used for all VPP tests. 3. Benchmarking - Added new processor micro-architectures: ARM/AArch64 (TaiShan) and Atom (Denverton). - New testbeds onboarded into FD.io CSIT CI/CD functional and performance test pipelines. - Daily trending with throughput changes monitoring, analytics and anomaly auto-detection. - Release reports with benchmarking data including throughput, latency, test repeatibility. - Updated CSIT benchmarking report specification - Consistent selection of tests across all testbeds and processor microarchitectures present in FD.io labs (Xeon, Atom, ARM) for iterative benchmarking tests to verify results repeatibility. Data presented in graphs conveying NDR (non-drop rate, zero packet loss) and PDR (partial drop rate) throughput statistics. Multi-core speedup and latency are also presented. - Consistent comparison of NDR and PDR throughput results across the releases. - Updated graph naming and test grouping to improve browsability and access to test data. - Increased test coverage in 2-node testbed environment (2n-skx). - Updated soak testing methodology and new results, aligned with latest IETF draft specification draft-vpolak-bmwg-plrsearch-02. 4. Infrastructure - API - PAPI (Python API) used for all VPP tests, migrated away from VAT (VPP API Test). - VPP API change detection and gating in VPP and CSIT CI/CD. - Test Environments - VPP functional device tests: migrated away from VIRL (VM based) to container environment (with Nomad). - Added new physical testbeds: ARM/AArch64 (TaiShan) and Atom (Denverton). - CSIT Framework - Conifguration keyword alignment across 2-node and 3-node testbeds to ease test portability across environments. - Installer - Updated bare-metal CSIT performance testbed installer (ansible). ------------------------------------------------------------------------ Points of Note in CSIT-1908 Report ------------------------------------------------------------------------ Indexed specific links listed at the bottom. 1. VPP release notes a. Changes in CSIT-1908: [1] b. Known issues: [2] 2. VPP performance - 64B/IMIX throughput graphs (selected NIC models): a. Graphs explained: [3] b. L2 Ethernet Switching: [4] c. IPv4 Routing: [5] d. IPv6 Routing: [6] e. SRv6 Routing: [7] f. IPv4 Tunnels: [8] g. KVM VMs vhost-user: [9] h. LXC/DRC Container Memif: [10] e. IPsec IPv4 Routing: [11] f. Virtual Topology System: [12] 3. VPP performance - multi-core and latency graphs: a. Speedup Multi-Core: [13] b. Latency: [14] 4. VPP system performance - NFV service density and TCP/IP: a. VNF (VM) Service Chains: [15] b. CNF (Container) Service Chains: [16] c. CNF (Container) Service Pipelines: [17] d. HTTP and TCP/IP: [18] 4. VPP performance comparisons a. VPP-19.08 vs. VPP-19.04: [19] 5. VPP performance test details - all NICs: a. Detailed results 64B IMIX 1518B 9kB: [20] b. Configuration: [21] DPDK Testpmd and L3fwd performance sections follow similar structure. 6. DPDK applications: a. Release notes: [22] b. DPDK performance - 64B throughput graphs: [23] c. DPDK performance - latency graphs: [24] d. DPDK performance - DPDK-1905 vs. DPDK-19.01: [25] Functional tests, including VPP_Device (functional device tests), VPP_VIRL and HoneyComb are all included in the report. Specific links within the report: [1] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/csit_release_notes.html#changes-in-csit-release [2] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/csit_release_notes.html#known-issues [3] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/index.html [4] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/l2.html [5] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/ip4.html [6] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/ip6.html [7] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/srv6.html [8] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/ip4_tunnels.html [9] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/vm_vhost.html [10] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/container_memif.html [11] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/ipsec.html [12] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_throughput_graphs/vts.html [13] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/throughput_speedup_multi_core/index.html [14] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/packet_latency_graphs/index.html [15] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/nf_service_density/vnf_service_chains.html [16] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/nf_service_density/cnf_service_chains.html [17] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/nf_service_density/cnf_service_pipelines.html [18] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/http_server_performance/index.html [19] https://docs.fd.io/csit/rls1908/report/vpp_performance_tests/comparisons/current_vs_previous_release.html [20] https://docs.fd.io/csit/rls1908/report/detailed_test_results/vpp_performance_results/index.html [21] https://docs.fd.io/csit/rls1908/report/test_configuration/vpp_performance_configuration/index.html [22] https://docs.fd.io/csit/rls1908/report/dpdk_performance_tests/csit_release_notes.html [23] https://docs.fd.io/csit/rls1908/report/dpdk_performance_tests/packet_throughput_graphs/index.html [24] https://docs.fd.io/csit/rls1908/report/dpdk_performance_tests/packet_latency_graphs/index.html [25] https://docs.fd.io/csit/rls1908/report/dpdk_performance_tests/comparisons/current_vs_previous_release.html ------------------------------------------------------------------------
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13967): https://lists.fd.io/g/vpp-dev/message/13967 Mute This Topic: https://lists.fd.io/mt/33142999/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
