On 08-Apr-19 3:38 PM, David Marchand wrote:
On Mon, Apr 8, 2019 at 4:03 PM Burakov, Anatoly
<anatoly.bura...@intel.com <mailto:anatoly.bura...@intel.com>> wrote:
On 08-Apr-19 2:58 PM, David Marchand wrote:
> On Mon, Apr 8, 2019 at 3:39 PM Burakov, Anatoly
> <anatoly.bura...@intel.com <mailto:anatoly.bura...@intel.com>
<mailto:anatoly.bura...@intel.com
<mailto:anatoly.bura...@intel.com>>> wrote:
>
> As a concrete proposal, my number one dream would be to see
> multiprocess
> gone. I also recall desire for "DPDK to be more lightweight",
and i
> maintain that DPDK *cannot* be lightweight if we are to support
> multiprocess - we can have one or the other, but not both.
However,
> realistically, i don't think dropping multiprocess is ever
going to
> happen - not only it is too entrenched in DPDK use cases, it is
> actually
> quite useful despite its flaws.
>
>
> Well, honestly, I'd like to hear about this.
> What are the real usecases for multi process support?
> Do we have even a single opensource project that uses it?
>
I'm aware of a few closed source usages of multiprocess. I also think
current versions of collectd rely on secondary process (there's been a
Telemetry API added to avoid that, but AFAIK the support for Telemetry
is not upstream in collectd yet), and so do/would any dump-style
applications - in fact, we ourselves include one such application in
our
codebase (pdump, proc-info, etc.).
Sorry, I don't want to highjack this thread, I can start a separate
thread if people feel like it.
If we go with stabilisation, we must be careful that we want to support
the features.
So about multiprocess, again, in those closed source projects you know
of, what are the usecases?
For what we provide in dpdk pdump, proc-info, referring to oneself is
not that convincing to me as I don't use those tools.
I don't see what we could not achieve the same with a control thread
running in the dpdk process and handling commands.
It would be open to the outside via a more standard channel, like a UNIX
socket or something like this.
If we need to declare a dynamic channel, it can be constructed as an
extension of the existing standard channel: we can open something like a
POSIX shm and push things in it.
Was this explored ?
There are certainly things that we can do that can make some aspects of
multiprocess redundant. For example, for any kind of collectd-like
scenario, the Telemetry API (or Keith's DFS, or...) could conceivably
provide a better and more maintainable way of doing things.
Our multiprocess also makes it easier to write pipeline/load-balancing
type applications. To see an example, look at our
multiprocess/client-server example. This is demonstrating how, instead
of writing one big monolithic application, one could instead write a
number of smaller applications each doing their thing. It is of course
possible to do the same without multiprocess, as evidenced by our sample
applications such as load-balancer, distributor, ip-pipeline etc., but
it is arguably easier to implement *real* applications that way due to
separation of concerns and more focused codebase.
However, there are two use cases that i can think of that are either
hard or outright not possible without our multiprocess API's. The first
one is dumping functionality. For example, dpdk_proc_info can display
info from a currently-running or defunct process - list its
memzones/mempools/etc. - basically, everything there is to know about
the shared memory can be known that way. While this isn't a "real" use
case, it is useful for debugging.
More importantly, our multiprocess model provides resilience. In an
event of a crash, the entire application is not brought down - instead,
only the crashed process goes down. It's not /perfect/ resilience, of
course, and there are caveats (memory leaking, locks, etc.), but you do
get /some/ resilience that way - your process went down, you spin
another secondary and you're back up and running again.
The above described scenario is how most people (that i know of) appear
to be using multiprocess - some kind of "crash-resilient"
load-balancing/pipelining app.
--
David Marchand
--
Thanks,
Anatoly
