> From: Mattias Rönnblom [mailto:hof...@lysator.liu.se] > Sent: Monday, 19 February 2024 15.32 > > On 2024-02-19 12:10, Morten Brørup wrote: > >> From: Mattias Rönnblom [mailto:hof...@lysator.liu.se] > >> Sent: Monday, 19 February 2024 08.49 > >> > >> On 2024-02-09 14:04, Morten Brørup wrote: > >>>> From: Mattias Rönnblom [mailto:hof...@lysator.liu.se] > >>>> Sent: Friday, 9 February 2024 12.46 > >>>> > >>>> On 2024-02-09 09:25, Morten Brørup wrote: > >>>>>> From: Mattias Rönnblom [mailto:mattias.ronnb...@ericsson.com] > >>>>>> Sent: Thursday, 8 February 2024 19.17 > >>>>>> > >>>>>> Introduce DPDK per-lcore id variables, or lcore variables for > >> short. > >>>>>> > >>>>>> An lcore variable has one value for every current and future > lcore > >>>>>> id-equipped thread. > >>>>>> > >>>>>> The primary <rte_lcore_var.h> use case is for statically > >> allocating > >>>>>> small chunks of often-used data, which is related logically, but > >>>> where > >>>>>> there are performance benefits to reap from having updates being > >>>> local > >>>>>> to an lcore. > >>>>>> > >>>>>> Lcore variables are similar to thread-local storage (TLS, e.g., > >> C11 > >>>>>> _Thread_local), but decoupling the values' life time with that > of > >>>> the > >>>>>> threads. > >>>>>> > >>>>>> Lcore variables are also similar in terms of functionality > >> provided > >>>> by > >>>>>> FreeBSD kernel's DPCPU_*() family of macros and the associated > >>>>>> build-time machinery. DPCPU uses linker scripts, which > effectively > >>>>>> prevents the reuse of its, otherwise seemingly viable, approach. > >>>>>> > >>>>>> The currently-prevailing way to solve the same problem as lcore > >>>>>> variables is to keep a module's per-lcore data as RTE_MAX_LCORE- > >>>> sized > >>>>>> array of cache-aligned, RTE_CACHE_GUARDed structs. The benefit > of > >>>>>> lcore variables over this approach is that data related to the > >> same > >>>>>> lcore now is close (spatially, in memory), rather than data used > >> by > >>>>>> the same module, which in turn avoid excessive use of padding, > >>>>>> polluting caches with unused data. > >>>>>> > >>>>>> Signed-off-by: Mattias Rönnblom <mattias.ronnb...@ericsson.com> > >>>>>> ---
[...] > > Ups... wrong reference! I meant to refer to _lcore_id, not > _thread_id. Correction: > > > > OK. I subconsciously ignored this mistake, and read it as "_lcore_id". :-) [...] > >> For DPDK modules using lcore variables and which treat unregistered > >> threads as "full citizens", I expect special handling of > unregistered > >> threads to be the norm. Take rte_random.h as an example. Current API > >> does not guarantee MT safety for concurrent calls of unregistered > >> threads. It probably should, and it should probably be by means of a > >> mutex (not spinlock). > >> > >> The reason I'm not running off to make a rte_random.c patch is > that's > >> it's unclear to me what is the role of unregistered threads in DPDK. > >> I'm > >> reasonably comfortable with a model where there are many threads > that > >> basically don't interact with the DPDK APIs (except maybe some very > >> narrow exposure, like the preemption-safe ring variant). One example > of > >> such a design would be big slow control plane which uses multi- > >> threading > >> and the Linux process scheduler for work scheduling, hosted in the > same > >> process as a DPDK data plane app. > >> > >> What I find more strange is a scenario where there are unregistered > >> threads which interacts with a wide variety of DPDK APIs, does so > >> at-high-rates/with-high-performance-requirements and are expected to > be > >> preemption-safe. So they are basically EAL threads without a lcore > id. > > > > Yes, this is happening in the wild. > > E.g. our application has a mode where it uses fewer EAL threads, and > processes more in non-EAL threads. So to say, the same work is > processed either by an EAL thread or a non-EAL thread, depending on the > application's mode. > > The extra array entry would be useful for such use cases. > > > > Is there some particular reason you can't register those non-EAL > threads? Legacy. I suppose we could just do that instead. Thanks for the suggestion! > > >> > >> Support for that latter scenario has also been voiced, in previous > >> discussions, from what I recall. > >> > >> I think it's hard to answer the question of a "unregistered thread > >> spare" for lcore variables without first knowing what the future > should > >> look like for unregistered threads in DPDK, in terms of being able > to > >> call into DPDK APIs, preemption-safety guarantees, etc. > >> > >> It seems that until you have a clearer picture of how generally to > >> treat > >> unregistered threads, you are best off with just a per-lcore id > >> instance > >> of lcore variables. > > > > I get your point. It also reduces the risk of bugs caused by > incorrect use of the additional entry. > > > > I am arguing for a different angle: Providing the extra entry will > help uncovering relevant use cases. > > > > Maybe have two "spares" in case you find two new uses cases? :) > > No, adding spares doesn't work, unless you rework the API and rename it > to fit the new purpose of not only providing per-lcore id variables, > but per-something-else. > OK. I'm convinced.
