After fixing few nits with Sergio (thanks for help) looks like we are able to measure multicore perf:
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-master-long/1200/console tc13-64B-2t2c-ethip4ipsecscale1ip4-ip4base-interfaces-aes-gcm-ndrdisc FINAL_RATE: 4668398.4375 pps (2x 2334199.21875 pps) This is linear scale if compare with previous 2.5Mpps. Thanks. Peter Mikus Engineer – Software Cisco Systems Limited From: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) Sent: Thursday, March 16, 2017 12:02 PM To: 'Sergio Gonzalez Monroy' <sergio.gonzalez.mon...@intel.com>; Rybalchenko, Kirill <kirill.rybalche...@intel.com>; Nicolau, Radu <radu.nico...@intel.com> Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco) <tifr...@cisco.com>; csit-...@lists.fd.io; vpp-dev <vpp-dev@lists.fd.io>; Maciek Konstantynowicz (mkonstan) <mkons...@cisco.com> Subject: RE: IPsec Multi-Tunnel performance test suite failure I modified the script to produce this config (2 threads): unix { nodaemon log /tmp/vpe.log cli-listen localhost:5002 full-coredump } api-trace { on } cpu { main-core 19 corelist-workers 20,21 } dpdk { socket-mem 1024,1024 dev default { num-rx-queues 1 } dev 0000:88:00.1 dev 0000:88:00.0 enable-cryptodev dev 0000:86:01.0 dev 0000:86:01.1 uio-driver igb_uio no-multi-seg } ip6 { hash-buckets 2000000 heap-size 3G } I am sure that QAT is initialized with 32VFs (cat /sys/bus/pci/devices/0000\:86\:00.0/sriov_numvfs show 32 on both machines), but: DUT1 is showing: vat# vat# DPDK Cryptodev support is disabled DUT2 is showing: vat# vat# worker crypto device id(type) 1 1(HW) 2 0(HW) The setup for both DUTs is the same. Do you know what may be wrong? In syslog I do see for both DUTs (lspci shows QAT initialized): … Mar 16 01:27:53 t3-sut1 kernel: [94934.751859] uio_pci_generic 0000:86:04.7: enabling device (0000 -> 0002) Mar 16 01:27:53 t3-sut1 kernel: [94934.751862] uio_pci_generic 0000:86:04.7: No IRQ assigned to device: no support for interrupts? … Mar 16 01:27:58 t3-sut2 kernel: [94846.606256] uio_pci_generic 0000:86:04.7: enabling device (0000 -> 0002) Mar 16 01:27:58 t3-sut2 kernel: [94846.606259] uio_pci_generic 0000:86:04.7: No IRQ assigned to device: no support for interrupts? So no more crypto not found. Any ideas? Peter Mikus Engineer – Software Cisco Systems Limited From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.mon...@intel.com] Sent: Wednesday, March 15, 2017 5:11 PM To: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) <pmi...@cisco.com<mailto:pmi...@cisco.com>> Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco) <tifr...@cisco.com<mailto:tifr...@cisco.com>>; csit-...@lists.fd.io<mailto:csit-...@lists.fd.io>; vpp-dev <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>>; Maciek Konstantynowicz (mkonstan) <mkons...@cisco.com<mailto:mkons...@cisco.com>>; Rybalchenko, Kirill <kirill.rybalche...@intel.com<mailto:kirill.rybalche...@intel.com>>; Nicolau, Radu <radu.nico...@intel.com<mailto:radu.nico...@intel.com>> Subject: Re: IPsec Multi-Tunnel performance test suite failure I reckon something like the following should do: enable-cryptodev dev 0000:86:01.0 dev 0000:86:01.1 Sergio On 15/03/2017 15:36, Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) wrote: So something like: enable-cryptodev dev 0000:86:01.0 enable-cryptodev dev 0000:86:02.0 ? Peter Mikus Engineer – Software Cisco Systems Limited From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.mon...@intel.com] Sent: Wednesday, March 15, 2017 4:14 PM To: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) <pmi...@cisco.com><mailto:pmi...@cisco.com>; Rybalchenko, Kirill <kirill.rybalche...@intel.com><mailto:kirill.rybalche...@intel.com>; Nicolau, Radu <radu.nico...@intel.com><mailto:radu.nico...@intel.com> Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco) <tifr...@cisco.com><mailto:tifr...@cisco.com>; csit-...@lists.fd.io<mailto:csit-...@lists.fd.io>; vpp-dev <vpp-dev@lists.fd.io><mailto:vpp-dev@lists.fd.io>; Maciek Konstantynowicz (mkonstan) <mkons...@cisco.com><mailto:mkons...@cisco.com> Subject: Re: IPsec Multi-Tunnel performance test suite failure My bad. I thought the test was already using two QAT VFs. Each workers needs one QAT VF. Sergio On 15/03/2017 13:47, Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) wrote: After I run CSIT with 2 physical cores and 2 worker-threads, the HW cryptodev is not working: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-master-all/1178/console Testing is HW is there was successful and it was initialized. Can you please take a look? The only change I did was adding 1 more worker threads. Initialization remains the same and Cryptodev HW was not recognized? Peter Mikus Engineer – Software Cisco Systems Limited From: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) Sent: Monday, March 13, 2017 2:19 PM To: 'Sergio Gonzalez Monroy' <sergio.gonzalez.mon...@intel.com><mailto:sergio.gonzalez.mon...@intel.com>; Maciek Konstantynowicz (mkonstan) <mkons...@cisco.com><mailto:mkons...@cisco.com>; Rybalchenko, Kirill <kirill.rybalche...@intel.com><mailto:kirill.rybalche...@intel.com>; Nicolau, Radu <radu.nico...@intel.com><mailto:radu.nico...@intel.com> Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco) <tifr...@cisco.com><mailto:tifr...@cisco.com>; csit-...@lists.fd.io<mailto:csit-...@lists.fd.io>; vpp-dev <vpp-dev@lists.fd.io><mailto:vpp-dev@lists.fd.io> Subject: RE: IPsec Multi-Tunnel performance test suite failure + Looping @csit-dev @vpp-dev I will add 2 workers/threads that is not a problem. To avoid possible exploding of number of test, we should pick only the representative one. Apart from implementation are there any other differences between tunnel and interface mode? Thanks. Peter Mikus Engineer – Software Cisco Systems Limited From: Sergio Gonzalez Monroy [mailto:sergio.gonzalez.mon...@intel.com] Sent: Monday, March 13, 2017 9:58 AM To: Maciek Konstantynowicz (mkonstan) <mkons...@cisco.com<mailto:mkons...@cisco.com>>; Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) <pmi...@cisco.com<mailto:pmi...@cisco.com>>; Rybalchenko, Kirill <kirill.rybalche...@intel.com<mailto:kirill.rybalche...@intel.com>>; Nicolau, Radu <radu.nico...@intel.com<mailto:radu.nico...@intel.com>> Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco) <tifr...@cisco.com<mailto:tifr...@cisco.com>> Subject: Re: IPsec Multi-Tunnel performance test suite failure First, thank you to all involved! I reckon those numbers are in the expected range. The current test is single thread with bidirectional traffic. I would definitely like to see tests with 2 workers/threads, one worker for each direction. One of the reasons is that we cannot saturate QAT with a single worker (QAT should be able to do +40Gbps of encryption). Would it make sense to have another set of tests with 2 workers or just update the current tests to use 2 workers? Regarding the difference between ipsec interface and tunnels (a.k.a. SPD), the results are expected. Basically, it is all about the SPD (Security Policy Database) implementation. The "tunnels" tests use the SPD, whereas the ipsec interfaces do not. The current SPD implementation in VPP follows the guidelines of the RFC, but it does not scale. The ipsec interfaces do not use the SPD at all and a single entry in the fib is all we need to "select" the traffic to encrypt. They effectively are different graph node paths, and even though both end up taking the same amount of cycles (at least for decryption), the interfaces scale much better. Sergio On 11/03/2017 18:36, Maciek Konstantynowicz (mkonstan) wrote: Great Peter, thanks for this final push ! Sergio, team - are these the results you expect to see? Why such a difference interfaces vs. tunnels at 1k scale? aes-gcm interfaces 1 tunnel 1k tunnels Mpps Gbps Mpps Gbps 64B 2.5 1.7 2.3 1.5 IMIX 2.4 7.1 2.1 6.4 1518B 2.4 28.9 2.1 26.0 cbc-sha1 interfaces 1 tunnel 1k tunnels Mpps Gbps Mpps Gbps 64B 2.5 1.7 2.3 1.5 IMIX 2.4 7.2 2.1 6.4 1518B 2.4 29.2 2.1 29.2 aes-gcm tunnels 1 tunnel 1k tunnels Mpps Gbps Mpps Gbps 64B 2.4 1.6 0.4 0.2 IMIX 2.4 7.0 0.3 1.0 1518B 2.2 27.8 0.4 4.3 cbc-sha1 tunnels 1 tunnel 1k tunnels Mpps Gbps Mpps Gbps 64B 2.5 1.7 0.4 0.3 IMIX 2.4 7.2 0.3 1.0 1518B 2.4 29.2 0.4 5.0 -Maciek On 11 Mar 2017, at 06:45, Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) <pmi...@cisco.com<mailto:pmi...@cisco.com>> wrote: IPSEC is now working. PDR and NDR results are same and can be found there: https://jenkins.fd.io/view/csit/job/csit-vpp-perf-master-all/1156/console Plots will be updated to display IPsecHW (seems like wrong xpath eval). I will check on Monday. So far I will run couple more iterations to see the results @Maciek, I think it is about time to populate all TBs with QAT. Can we coordinate? Peter Mikus Engineer – Software Cisco Systems Limited From: Nicolau, Radu [mailto:radu.nico...@intel.com] Sent: Wednesday, March 08, 2017 2:07 PM To: Gonzalez Monroy, Sergio <sergio.gonzalez.mon...@intel.com<mailto:sergio.gonzalez.mon...@intel.com>>; Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) <pmi...@cisco.com<mailto:pmi...@cisco.com>>; Rybalchenko, Kirill <kirill.rybalche...@intel.com<mailto:kirill.rybalche...@intel.com>> Cc: Tibor Frank -X (tifrank - PANTHEON TECHNOLOGIES at Cisco) <tifr...@cisco.com<mailto:tifr...@cisco.com>>; Maciek Konstantynowicz (mkonstan) <mkons...@cisco.com<mailto:mkons...@cisco.com>> Subject: RE: IPsec Multi-Tunnel performance test suite failure Hi, I submitted a small patch to only bind QAT VFs. https://gerrit.fd.io/r/#/c/5671/ The downside is that the additional check will have to be updated for new devices. Regards, Radu
_______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
