> From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Anatoly Burakov > Sent: Friday, May 19, 2017 5:40 PM > To: dev@dpdk.org > Subject: [dpdk-dev] [RFC 0/4] DPDK multiprocess rework > > This is a proof-of-concept proposal for rework of how DPDK secondary > processes work. While the code has some limitations, it works well enough to > demonstrate the concept, and it can successfully run all existing multiprocess > applications. > > Current problems with DPDK secondary processes: > * ASLR interferes with mappings > * "Fixed" by disabling ASLR, but not really a solution > * Secondary process may map things into where we want to map shared > memory > * _Almost_ works with --base-virtaddr, but unreliable and tedious > * Function pointers don't work (so e.g. hash library is broken) > > Proposed solution: > > Instead of running secondary process and mapping resources from primary > process, the following is done: > 0) compile all applications as position-indendent executables, compile DPDK > as > a shared library > 1) fork() from primary process > 2) dlopen() secondary process binary > 3) use dlsym() to find entry point > 4) run the application code while having all resources already mapped > > Benefits: > * No more ASLR issues > * No need for --base-virtaddr > * Function pointers from primary process will work in secondaries > * Hash library (and any other library that uses function pointers > internally) > will work correctly in multi-process scenario > * ethdev data can be moved to shared memory > * Primary process interrupt callbacks can be run by secondary process > * More secure as all applications are compiled as position-indendent binaries > (default on Fedora) > > Potential drawbacks (that we could think of): > * Kind of a hack > * Puts some code restrictions on secondary processes > * Anything happening before EAL init will be run twice > * Some use cases are no longer possible (attaching to a dead primary) > * May impact binaries compiled to use a lot (kilobytes) of thread-local > storage[1] > * Likely wouldn't work for static linking > > There are also a number of issues that need to be resolved, but those are > implementation details and are out of scope for RFC. > > What is explicitly out of scope: > * Fixing interrupts in secondary processes > * Fixing hotplug in secondary processes > > These currently do not work in secondary processes, and this proposal does > nothing to change that. They are better addressed using dedicated EAL- > internal IPC proposal. > > > Technical nitty-gritty > > Things quickly get confusing, so terminology: > - Original Primary is normal DPDK primary process > - Forked Primary is a "clean slate" primary process, from which all secondary > processes will be forked (threads and fork don't mix well, so fork is done > after all the hugepage and PCI data is mapped, but before all the threads > are > spun up) > - Original Secondary is a process that connects to Forked Primary, sends > some > data and and triggers a fork > - Forked Secondary is _actual_ secondary process (forked from Forked > Primary) > > Timeline: > - Original Primary starts > - Forked Primary is forked from Original Primary > - Original Secondary starts and connects to Forked Primary > - Forked Primary forks into Forked Secondary > - Original Secondary waits until Forked Secondary dies > > During EAL init, Original Primary does a fork() to form a Forked Primary - a > "clean slate" starting point for secondary processes. Forked Primary opens a > local socket (a-la VFIO) and starts listening for incoming connections. > > Original Secondary process connects to Forked Primary, sends stdout/log > fd's, command line parameters, etc. over local socket, and sits around waiting > for Forked Secondary to die, then exits (Original Secondary does _not_ map > anything or do any EAL init, it rte_exit()'s from inside rte_eal_init()). > Forked > Secondary process then executes main(), passing all command-line > arguments, and execution of secondary process resumes. > > Why pre-fork and not pthread like VFIO? > > Pthreads and fork() don't mix well, because fork() stops the world (all > threads disappear, leaving behind thread stacks, locks and possibly > inconsistent state of both app data and system libraries). On the other hand, > forking from single- threaded context is safe. Current implementation > doesn't _exactly_ fork from a single-threaded context, but this can be fixed > later by rearranging EAL init. > > [1]: https://www.redhat.com/archives/phil-list/2003- > February/msg00077.html >
Ping
