I was assuming socket_id == numa, so not the issue there.
Are the crypto devices also in each numa node?
On Mon, Feb 4, 2019 at 11:22 PM Roberts, Lee A. <lee.robe...@hpe.com> wrote:
> Sergio,
>
>
>
> The corelist-workers do translate to single cores in each of the NUMA
> zones,
>
> as shown below:
>
>
>
> cpu {
>
> main-core 1
>
> corelist-workers 2,6,10,14
>
> }
>
>
>
>
>
> [root@cl41 ~]# numactl -H
>
> available: 4 nodes (0-3)
>
> node 0 cpus: 0 1 2 3
>
> node 0 size: 32656 MB
>
> node 0 free: 12133 MB
>
> node 1 cpus: 4 5 6 7
>
> node 1 size: 32767 MB
>
> node 1 free: 11632 MB
>
> node 2 cpus: 8 9 10 11
>
> node 2 size: 32767 MB
>
> node 2 free: 13486 MB
>
> node 3 cpus: 12 13 14 15
>
> node 3 size: 32767 MB
>
> node 3 free: 14229 MB
>
> node distances:
>
> node 0 1 2 3
>
> 0: 10 16 16 16
>
> 1: 16 10 16 16
>
> 2: 16 16 10 16
>
> 3: 16 16 16 10
>
> [root@cl41 ~]#
>
>
>
> Might there be confusion between “NUMA node” and “socket_id” definitions?
>
> My configuration file assumes that “socket_id” ==> NUMA, but this is
> actually
>
> a single-socket system with four NUMA zones.
>
>
>
> - Lee
>
>
>
>
>
> *From:* Sergio G.M. [mailto:sgmon...@gmail.com]
> *Sent:* Monday, February 04, 2019 2:55 PM
> *To:* Roberts, Lee A. <lee.robe...@hpe.com>
> *Cc:* manuel.alo...@cavium.com; vpp-dev@lists.fd.io
> *Subject:* Re: [vpp-dev] Question about crypto dev queue pairs #vpp
>
>
>
> Hi Lee,
>
>
>
> Looking at your startup.conf below, are you sure those parameters are
> correct? are you running 4 numa node platform?
>
> If that is the case, you should also use cores from each numa node.
>
>
>
> vdev crypto_ccp0,socket_id=0
>
> vdev crypto_ccp1,socket_id=1
>
> vdev crypto_ccp2,socket_id=2
>
> vdev crypto_ccp3,socket_id=3
>
>
>
> I think the auto mapping is trying to map core and crypto devices from the
> same numa node.
>
>
>
> On Fri, Feb 1, 2019 at 10:16 PM Roberts, Lee A. <lee.robe...@hpe.com>
> wrote:
>
> Sergio,
>
>
>
> I’m having partial success. I can get a single “vdev crypto_ccp” to be
>
> recognized with a single worker core, but I’m not able to get multiple
>
> devices.
>
>
>
> In my /etc/vpp/startup.conf, I have the following:
>
>
>
> ...
>
> cpu {
>
> main-core 1
>
> corelist-workers 2,6,10,14
>
> }
>
>
>
> dpdk {
>
> socket-mem 2048,2048,2048,2048
>
> dev default {
>
> num-rx-queues 2
>
> num-tx-queues 2
>
> num-tx-desc 2048
>
> num-rx-desc 2048
>
> }
>
> uio-driver uio-pci-generic
>
> dev 0000:41:00.0
>
> dev 0000:41:00.1
>
> vdev crypto_ccp0,socket_id=0
>
> vdev crypto_ccp1,socket_id=1
>
> vdev crypto_ccp2,socket_id=2
>
> vdev crypto_ccp3,socket_id=3
>
> }
>
> ...
>
>
>
> When I start VPP using this configuration, I see a message about “not
> enough DPDK crypto resources,
>
> default to OpenSSL”:
>
>
>
> Feb 1 13:52:38 cl41 vpp: /usr/bin/vpp[15173]: dpdk_config:1344:
> socket-mem argument is deprecated
>
> Feb 1 13:52:38 cl41 vpp: /usr/bin/vpp[15173]: dpdk: EAL init args: -c
> 4446 -n 4 --in-memory --vdev crypto_ccp0,socket_id=0 --vdev
> crypto_ccp1,socket_id=1 --vdev crypto_ccp2,socket_id=2 --vdev
> crypto_ccp3,socket_id=3 --file-prefix vpp -w 0000:41:00.0 -w 0000:41:00.1
> --master-lcore 1
>
> Feb 1 13:52:38 cl41 /usr/bin/vpp[15173]: dpdk: EAL init args: -c 4446 -n
> 4 --in-memory --vdev crypto_ccp0,socket_id=0 --vdev
> crypto_ccp1,socket_id=1 --vdev crypto_ccp2,socket_id=2 --vdev
> crypto_ccp3,socket_id=3 --file-prefix vpp -w 0000:41:00.0 -w 0000:41:00.1
> --master-lcore 1
>
> Feb 1 13:52:38 cl41 vnet[15173]: dpdk_ipsec_process:1014: not enough
> DPDK crypto resources, default to OpenSSL
>
> Feb 1 13:53:13 cl41 vnet[15173]: unix_signal_handler:159: received
> signal SIGCONT, PC 0x7f9aa76687e0
>
> Feb 1 13:53:13 cl41 systemd: Stopping Vector Packet Processing Process...
>
> Feb 1 13:53:13 cl41 vnet[15173]: received SIGTERM, exiting...
>
> Feb 1 13:53:13 cl41 vnet[15173]: unix_signal_handler:159: received
> signal SIGCONT, PC 0x7f9aa76687e0
>
> Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1456
>
> Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1468
>
> Feb 1 13:53:13 cl41 vpp: Queue 0 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 1 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 2 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 3 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 4 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1456
>
> Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1468
>
> Feb 1 13:53:13 cl41 vpp: Queue 0 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 1 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 2 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 3 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 4 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1456
>
> Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1468
>
> Feb 1 13:53:13 cl41 vpp: Queue 0 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 1 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 2 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 3 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 4 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1456
>
> Feb 1 13:53:13 cl41 vpp: CCP : Detected CCP device with ID = 0x1468
>
> Feb 1 13:53:13 cl41 vpp: Queue 0 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 1 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 2 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 3 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: Queue 4 can access 7 LSB regions of mask 254
>
> Feb 1 13:53:13 cl41 vpp: CCP : Crypto device count = 4
>
> Feb 1 13:53:13 cl41 systemd: Stopped Vector Packet Processing Process.
>
>
>
>
>
> If I modify the startup.conf to only have one worker core (“corelist-workers
> 2”),
>
> I don’t get the message about “not enough DPDK crypto resources”, and I
> see:
>
>
>
> vpp# show dpdk crypto device
>
> crypto_ccp0 crypto_ccp up
>
> numa_node 0, max_queues 1
>
> free_resources 0, used_resources 1
>
> SYMMETRIC_CRYPTO, SYM_OPERATION_CHAINING, HW_ACCELERATED
>
> Cipher: aes-cbc-128, aes-cbc-192, aes-cbc-256, aes-ctr-128, aes-ctr-192,
> aes-ctr-256, aes-gcm-128, aes-gcm-192, aes-gcm-256
>
> Auth:
>
>
>
> vpp#
>
>
>
>
>
> When I look at the source code in crypto_parse_capabilities(), I am
> concerned about the lines
>
> “ dcm->enabled |= (alg->resources >= n_mains);” in the
> following code:
>
>
>
> static void
>
> crypto_parse_capabilities (crypto_dev_t * dev,
>
> const struct rte_cryptodev_capabilities *cap,
>
> u32 n_mains)
>
> {
>
> dpdk_crypto_main_t *dcm = &dpdk_crypto_main;
>
> crypto_alg_t *alg;
>
> u8 len, inc;
>
>
>
> for (; cap->op != RTE_CRYPTO_OP_TYPE_UNDEFINED; cap++)
>
> {
>
> /* A single capability maps to multiple cipher/auth algorithms */
>
> switch (cap->sym.xform_type)
>
> {
>
> case RTE_CRYPTO_SYM_XFORM_AEAD:
>
> case RTE_CRYPTO_SYM_XFORM_CIPHER:
>
> inc = cap->sym.cipher.key_size.increment;
>
> inc = inc ? inc : 1;
>
> for (len = cap->sym.cipher.key_size.min;
>
> len <= cap->sym.cipher.key_size.max; len += inc)
>
> {
>
> alg = cipher_cap_to_alg (cap, len);
>
> if (!alg)
>
> continue;
>
> dev->cipher_support[cipher_alg_index (alg)] = 1;
>
> alg->resources += vec_len (dev->free_resources);
>
> /* At least enough resources to support one algo */
>
> dcm->enabled |= (alg->resources >= n_mains);
>
> }
>
> break;
>
> case RTE_CRYPTO_SYM_XFORM_AUTH:
>
> inc = cap->sym.auth.digest_size.increment;
>
> inc = inc ? inc : 1;
>
> for (len = cap->sym.auth.digest_size.min;
>
> len <= cap->sym.auth.digest_size.max; len += inc)
>
> {
>
> alg = auth_cap_to_alg (cap, len);
>
> if (!alg)
>
> continue;
>
> dev->auth_support[auth_alg_index (alg)] = 1;
>
> alg->resources += vec_len (dev->free_resources);
>
> /* At least enough resources to support one algo */
>
> dcm->enabled |= (alg->resources >= n_mains);
>
> }
>
> break;
>
> default:
>
> ;
>
> }
>
> }
>
> }
>
>
>
> crypto_scan_devs() calls crypto_parse_capabilities() within a loop over
> cryptodevs.
>
> The code appears to be comparing the resources of each cryptodev to the
> total number,
>
> n_mains. Should the code be comparing the sum of resources (from all of
> the cryptodevs)
>
> to the total value, n_mains?
>
>
>
> - Lee
>
>
>
>
>
> *From:* vpp-dev@lists.fd.io [mailto:vpp-dev@lists.fd.io] *On Behalf Of *Sergio
> Gonzalez Monroy
> *Sent:* Wednesday, January 30, 2019 1:59 PM
> *To:* Roberts, Lee A. <lee.robe...@hpe.com>; manuel.alo...@cavium.com;
> vpp-dev@lists.fd.io
> *Subject:* Re: [vpp-dev] Question about crypto dev queue pairs #vpp
>
>
>
> Hi folks,
>
>
>
> Give the following patch a try and let me know.
>
> https://gerrit.fd.io/r/#/c/17184/
>
>
>
> I have done some very basic testing with SW cryptodev with single queue
> pair and all seems fine.
>
>
>
> Let me know if there is any issue with it.
>
>
>
> Regards,
>
> Sergio
> ------------------------------
>
> *From:* Roberts, Lee A. <lee.robe...@hpe.com>
> *Sent:* Tuesday, January 29, 2019 3:49 PM
> *To:* Sergio Gonzalez Monroy; manuel.alo...@cavium.com;
> vpp-dev@lists.fd.io
> *Subject:* RE: [vpp-dev] Question about crypto dev queue pairs #vpp
>
>
>
> Sergio,
>
>
>
> I encountered the same problem when attempting to enable the AMD CCP poll
> mode driver
>
> in VPP 18.10.
>
>
>
> As mentioned earlier in this e-mail thread, with max_qp = 1, max_res_idx
> becomes 65535
>
> in the following statement:
>
>
>
> max_res_idx = (dev->max_qp / 2) - 1;
>
>
>
> I hadn’t found the time to study/debug the code to understand whether this
> was a
>
> VPP or DPDK issue. If you have a patch available in the next few days, I
> could
>
> test it with the AMD CCP device.
>
>
>
> - Lee Roberts
>
>
>
>
>
> *From:* vpp-dev@lists.fd.io [mailto:vpp-dev@lists.fd.io
> <vpp-dev@lists.fd.io>] *On Behalf Of *Sergio Gonzalez Monroy
> *Sent:* Tuesday, January 29, 2019 2:18 AM
> *To:* manuel.alo...@cavium.com; vpp-dev@lists.fd.io
> *Subject:* Re: [vpp-dev] Question about crypto dev queue pairs #vpp
>
>
>
> Hi Manuel,
>
>
>
> This is likely a mismatch in VPP side. I only tested it with QAT (2 qps
> per VF) and SW cryptodevs (default 8 qps) at the time (over a year ago).
>
>
>
> I only tested it with SW cryptodevs and QAT, that was the HW I had access
> to.
>
>
>
> So like I mentioned before, if you do not want to rework the code to
> support 1 qp per resource, then a check for at least 2 qps per device is
> required to use that device.
>
>
>
> I could provide a patch to use 1 pq per resource over the next few days if
> you are interested in it or could review if you decide to do the work.
>
>
>
> Which device do you want to use?
>
>
>
> Regards,
>
> Sergio
> ------------------------------
>
> *From:* vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> on behalf of
> manuel.alo...@cavium.com <manuel.alo...@cavium.com>
> *Sent:* Monday, January 28, 2019 4:15 PM
> *To:* vpp-dev@lists.fd.io
> *Subject:* Re: [vpp-dev] Question about crypto dev queue pairs #vpp
>
>
>
> Hi Sergio,
>
> thank you for the explanation, I see that there are 2 (or more qps). My
> concern was due to dpdk, since there are a few device drivers exporting
> only one queue pair for their crypto devices.
> (I followed the code assuming one qps, based on a dpdk-18.11 exported
> value)
> So I do not know where is the mismatching, vpp or dpdk?
>
>
> BR,
> Manuel
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#12172): https://lists.fd.io/g/vpp-dev/message/12172
Mute This Topic: https://lists.fd.io/mt/29538345/21656
Mute #vpp: https://lists.fd.io/mk?hashtag=vpp&subid=1480452
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
