>-----Original Message-----
>From: Ilya Maximets <i.maxim...@ovn.org>
>Sent: Tuesday, 9 May 2023 21:04
>To: Eli Britstein <el...@nvidia.com>; ovs-discuss@openvswitch.org
>Cc: Kevin Traynor <ktray...@redhat.com>; Aaron Conole
><acon...@redhat.com>; i.maxim...@ovn.org
>Subject: Re: [ovs-discuss] dpdk-lcore-mask
>
>External email: Use caution opening links or attachments
>
>
>On 5/5/23 12:21, Eli Britstein via discuss wrote:
>> Hello,
>>
>> With [1], dpdk-lcore-mask configuration was introduced, instead of passing
>explicitly dpdk EAL options.
>> Commits [2]-[3] handle this code area.
>>
>> Currently, the affinity is restored to the original one only if the user did
>> not
>set any dpdk-lcore-mask, and it is "auto_determine".
>> In case it has been set, the affinity is not restored, so the main thread's
>affinity is for a single core, dpdk's "main core".
>> When other threads, such as "hw-offload" threads, "ct_clean" etc., they
>inherit this affinity, thus scaling doesn't work properly.
>>
>> Also, DPDK create worker threads, but they are not in use anyway by OVS.
>> PMD threads are created and registered to dpdk as "non-EAL" threads, and
>their affinity is managed by OVS according to "pmd-cpu-mask" configuration.
>> They get with an rte_lcore_id after the EAL IDs. Those are in use for
>example for mempool cache access, but that's OK as dpdk's cache is always up
>to RTE_MAX_LCORE.
>>
>> I wonder what the use is anyway for configuring dpdk-lcore-mask.
>> I think the affinity should always be restored, not only in the
>"auto_determine" case, but even so, there are still the worker threads
>created for vain.
>> Is there a use-case I am missing?
>
>There is a variety of other threads in dpdk that handle "interrupts", alarms,
>vhost events. And these threads are getting their affinity form the lcore-mask
>as well, IIRC.
That is correct. Those are "control threads" that are created by dpdk inside
rte_eal_init.
Their affinity is set by default to other cores than ones in -c. See those:
http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#eal-pthread-and-lcore-affinity
4.3.4. Control Thread API
It is calculated in compute_ctrl_threads_cpuset().
DPDK support setting the affinity of those threads by "-s" option, but it is
not exposed in OVS.
See
https://doc.dpdk.org/guides/freebsd_gsg/freebsd_eal_parameters.html
>
>In general, dpdk-lcore-mask config is not particularly useful indeed, so if you
>don't have a use case for it, then it likely shouldn't be set.
>
>Regarding restoration logic. It seems to come from this comment:
>
>https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatch
>work.ozlabs.org%2Fproject%2Fopenvswitch%2Fpatch%2F1450463278-7931-4-
>git-send-email-
>aconole%40redhat.com%2F%231222040&data=05%7C01%7Celibr%40nvidia.co
>m%7C06aacf4aadf4486d4bf608db50b7c768%7C43083d15727340c1b7db39efd9
>ccc17a%7C0%7C0%7C638192522479282270%7CUnknown%7CTWFpbGZsb3d8e
>yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
>%7C3000%7C%7C%7C&sdata=x4RpaN6B1xfxQr2DZ1%2Br%2BBIbLSfCGeFncM
>BFM7k5kdI%3D&reserved=0
>
>The idea appears to be that users may want to run OVS completely on isolated
>cores and provide them via lcore-mask. In this case we should not restore the
>affinity.
The affinity was always restored in [2], and restricted only to the
"auto_determine" case by [3].
Currently, if the user sets dpdk-lcore-mask:
1. DPDK's control threads are created with affinity of ~mask.
2. EAL threads are created with affinity of the mask, however with no use in
OVS.
3. OVS main thread's affinity is not restored and has the affinity of the
"main-lcore".
4. Other OVS threads created (ct_clean, hw_offload etc.) inherit the sole
affinity of the main thread (set to "main-lcore") which is wrong behavior.
If the affinity is always restored, it will resolve #3,#4 but still have #1,#2.
If this configuration is completely removed, all 1-4 will be resolved.
Unless I missed something, it is better to completely remove this argument
(dpdk-lcore-mask).
What do you think?
>
>CC: Kevin and Aaron.
>
>Best regards, Ilya Maximets.
>
>>
>> Thanks,
>> Eli
>>
>> [1] bab694097133 ("netdev-dpdk: Convert initialization from cmdline to
>> db") [2] 563c98d86e77 ("netdev-dpdk: Restore thread affinity after
>> DPDK init") [3] 88964e6428dc ("netdev-dpdk: Autofill lcore coremask if
>absent")
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
