> -----Original Message----- > From: Jerin Jacob <jerinjac...@gmail.com> > Sent: Tuesday, June 13, 2023 7:13 PM > To: Yan, Zhirun <zhirun....@intel.com> > Cc: dev@dpdk.org; jer...@marvell.com; kirankum...@marvell.com; > ndabilpu...@marvell.com; step...@networkplumber.org; > pbhagavat...@marvell.com; david.march...@redhat.com; Liang, Cunming > <cunming.li...@intel.com>; Wang, Haiyue <haiyue.w...@intel.com>; > mattias.ronnblom <mattias.ronnb...@ericsson.com> > Subject: Re: [PATCH v13 00/16] graph enhancement for multi-core dispatch > > On Tue, Jun 13, 2023 at 3:53 PM Zhirun Yan <zhirun....@intel.com> wrote: > > > > V13: > > Add sub header into meson indirect_headers list to export.(change > > meson.build in patch02, 05) > > Please rebase with dpdk.org/main. There are build issues now also make sure > monitor CI after pushing the patches. > Got it. I will rebase and fix the issues.
> [569/2998] Compiling C object lib/librte_node.a.p/node_ethdev_ctrl.c.o > [570/2998] Compiling C object lib/librte_graph.a.p/graph_graph.c.o > [571/2998] Compiling C object lib/librte_node.a.p/node_ip6_rewrite.c.o > FAILED: lib/librte_node.a.p/node_ip6_rewrite.c.o > ccache gcc -Ilib/librte_node.a.p -Ilib -I../lib -Ilib/node -I../lib/node -I. > -I.. -Iconfig > -I../config -Ilib/eal/include -I../lib/eal/include -Ilib/eal/linux/include - > I../lib/eal/linux/include -Ilib/eal/x86/include -I../lib/eal/x86/include - > Ilib/eal/common -I../lib/eal/common -Ilib/eal -I../lib/eal -Ilib/kvargs - > I../lib/kvargs -Ilib/metrics -I../lib/metrics -Ilib/telemetry > -I../lib/telemetry - > Ilib/graph -I../lib/graph -Ilib/pcapng -I../lib/pcapng -Ilib/ethdev > -I../lib/ethdev - > Ilib/net -I../lib/net -Ilib/mbuf -I../lib/mbuf -Ilib/mempool -I../lib/mempool > - > Ilib/ring -I../lib/ring -Ilib/meter -I../lib/meter -Ilib/lpm -I../lib/lpm > -Ilib/hash - > I../lib/hash -Ilib/rcu -I../lib/rcu -Ilib/cryptodev -I../lib/cryptodev -fd > iagnostics- > color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Werror - > O2 -g -include rte_config.h -Wcast-qual -Wdeprecated -Wformat -Wformat- > nonliteral -Wformat-security -Wmissing-declarations -Wmissing-prototypes - > Wnested-ext erns -Wold-style-definition -Wpointer-arith -Wsign-compare - > Wstrict-prototypes -Wundef -Wwrite-strings -Wno-address-of-packed-member > -Wno-packed-not-aligned -Wno-missing-field-initializers -Wno-zero-length- > bounds -D_GNU_SOURCE -fPIC -march= native -DALLOW_EXPERIMENTAL_API - > DALLOW_INTERNAL_API -Wno-format-truncation -fno-strict-aliasing - > DRTE_LOG_DEFAULT_LOGTYPE=lib.node -MD -MQ > lib/librte_node.a.p/node_ip6_rewrite.c.o -MF > lib/librte_node.a.p/node_ip6_rewrite.c.o.d -o lib/librt > e_node.a.p/node_ip6_rewrite.c.o -c ../lib/node/ip6_rewrite.c > ../lib/node/ip6_rewrite.c:8:10: fatal error: rte_graph_worker.h: No such file > or > directory > 8 | #include <rte_graph_worker.h> > | ^~~~~~~~~~~~~~~~~~~~ > compilation terminated. > [572/2998] Compiling C object > drivers/libtmp_rte_common_cnxk.a.p/common_cnxk_roc_idev.c.o > > > > > > V12: > > Fix compilation broken at patch 1.(keep renamed header align with > > patch 1,2) > > > > V11: > > Update comments and fix to add experimental flags for > rte_graph_model_is_valid() in patch 04. > > Update added symbols in alphabetical order in version.map with patch > 04,05,06,08,10. > > Update commit message in patch 16. > > > > V10: > > Add rte_graph_worker_model_no_check_get() for fast path, extract > > rte_graph_model_is_valid() in patch 04. > > Change RTE_ASSERT to return in patch 06. > > Change to treat not defined RTE_GRAPH_MODEL_SELECT as runtime pick in > patch 13. > > Move stats into dispatch union in patch 14. > > Change example to align with RTE_GRAPH_MODEL_SELECT scheme in patch > 16. > > Squash patch 17(doc) into patch 13(prog_guide), 16(example guide). > > > > > > V9: > > Fix CI build issues for doc building(move TAILQ next pointer out of union) > > in > patch 09,10. > > Fix graph model check in rte_graph_worker_model_set() in patch 04. > > Fix typo in doc. > > > > V8: > > No performance dorp for original l3fwd-graph and graph_perf_autotest. > > > > Update graph model set/get functions and add graph_model_is_valid() in > patch 04. > > Update doc for new scheme usage(choose model in runtime or compile time). > > Update dispatch schedule struct into union. > > Change enum rte_graph_worker_model to macro define in > rte_graph_worker_common.h. > > Add model clone in graph_clone() in patch 08. > > Remove unnecessary inline for slow path func graph_src_node_avail() in patch > 06. > > > > > > V7: > > Revert rte_rdtsc_precise() in fastpath to fix performance issues in patch > > 03. > > Introduce new scheme for model choosing. Use RTE_GRAPH_MODEL_SELECT > to choose in > > compile-time in patch 13, 15.(must have rte_graph_worker_model_set() to > help > > other config func to do model specific things like alloc wq, collect > > stats) Extract the common func clone_name() into graph_private.h for > graph/node clone in > > patch 07.(new patch) > > Use rte_graph->model in rte_graph_worker_model_set() instead of > RTE_PER_LCORE_*. > > Add test case for all new APIs in patch 16(new patch). > > Remove *_END line in enum rte_graph_worker_model in patch 04. > > Add model check for graph lcore binding. > > Rename workqueue as graph_mcore_dispatch_wq_node in patch 09. > > Change all new model files/APIs with prefix _mcore_dispatch_. > > Change description of new API, comments of func/structure to explicitly > mention for > > mcore dispatch model only. Add Doxygen comments. > > Update l3fwd-graph with new scheme, Update doc. > > Update MAINTAINERS. > > Fix typo and format issues. > > > > V6: > > Change rte_rdtsc() to rte_rdtsc_precise(). > > Add union in rte_graph_param to configure models. > > Remove memset in fastpath, add RTE_ASSERT for cloned graph. > > Update copyright in patch 02. > > Update l3fwd-graph node affinity, start from rx core successively. > > > > V5: > > Fix CI build issues about dynamically update doc. > > > > V4: > > Fix CI build issues about undefined reference of sched apis. > > Remove inline for model setting. > > > > V3: > > Fix CI build issues about TLS and typo. > > > > V2: > > Use git mv to keep git history. > > Use TLS for per-thread local storage. > > Change model name to mcore dispatch. > > Change API with specific mode name. > > Split big patch. > > Fix CI issues. > > Rebase l3fwd-graph example. > > Update doc and maintainers files. > > > > > > Currently, rte_graph supports RTC (Run-To-Completion) model within > > each of a single core. > > RTC is one of the typical model of packet processing. Others like > > Pipeline or Hybrid are lack of support. > > > > The patch set introduces a 'multicore dispatch' model selection which > > is a self-reacting scheme according to the core affinity. > > The new model enables a cross-core dispatching mechanism which employs > > a scheduling work-queue to dispatch streams to other worker cores > > which being associated with the destination node. When core flavor of > > the destination node is a default 'current', the stream can be > > continue executed as normal. > > > > Example: > > 3-node graph targets 3-core budget > > > > RTC: > > Graph: node-0 -> node-1 -> node-2 @Core0. > > > > + - - - - - - - - - - - - - - - - - - - - - + > > ' Core #0/1/2 ' > > ' ' > > ' +--------+ +---------+ +--------+ ' > > ' | Node-0 | --> | Node-1 | --> | Node-2 | ' > > ' +--------+ +---------+ +--------+ ' > > ' ' > > + - - - - - - - - - - - - - - - - - - - - - + > > > > Dispatch: > > > > Graph topo: node-0 -> Core1; node-1 -> node-2; node-2 -> node-3. > > Config graph: node-0 @Core0; node-1/3 @Core1; node-2 @Core2. > > > > .. code-block:: diff > > > > + - - - - - -+ +- - - - - - - - - - - - - + + - - - - - -+ > > ' Core #0 ' ' Core #1 ' ' Core #2 ' > > ' ' ' ' ' ' > > ' +--------+ ' ' +--------+ +--------+ ' ' +--------+ ' > > ' | Node-0 | - - - ->| Node-1 | | Node-3 |<- - - - | Node-2 | ' > > ' +--------+ ' ' +--------+ +--------+ ' ' +--------+ ' > > ' ' ' | ' ' ^ ' > > + - - - - - -+ +- - -|- - - - - - - - - - + + - - -|- - -+ > > | | > > + - - - - - - - - - - - - - - - - + > > > > > > The patch set has been break down as below: > > > > 1. Split graph worker into common and default model part. > > 2. Inline graph node processing to make it reusable. > > 3. Add set/get APIs to choose worker model. > > 4. Introduce core affinity API to set the node run on specific worker core. > > (only use in new model) > > 5. Introduce graph affinity API to bind one graph with specific worker > > core. > > 6. Introduce graph clone API. > > 7. Introduce stream moving with scheduler work-queue in patch 8~12. > > 8. Add stats for new models. > > 9. Abstract default graph config process and integrate new model into > > example/l3fwd-graph. Add new parameters for model choosing. > > > > We could run with new worker model by this: > > ./dpdk-l3fwd-graph -l 8,9,10,11 -n 4 -- -p 0x1 --config="(0,0,9)" -P > > --model="dispatch" > > > > References: > > https://static.sched.com/hosted_files/dpdkuserspace22/a6/graph%20intro > > duce%20remote%20dispatch%20for%20mult-core%20scaling.pdf > > > > > > > > > > Zhirun Yan (16): > > graph: rename rte_graph_work as common > > graph: split graph worker into common and default model > > graph: move node process into inline function > > graph: add get/set graph worker model APIs > > graph: introduce graph node core affinity API > > graph: introduce graph bind unbind API > > graph: move node clone name func into private as common > > graph: introduce graph clone API for other worker core > > graph: add structure for stream moving between cores > > graph: introduce stream moving cross cores > > graph: enable create and destroy graph scheduling workqueue > > graph: introduce graph walk by cross-core dispatch > > graph: enable graph multicore dispatch scheduler model > > graph: add stats for mcore dispatch model > > test/graph: add functional tests for mcore dispatch model > > examples/l3fwd-graph: introduce mcore dispatch worker model > > > > MAINTAINERS | 3 +- > > app/test/test_graph.c | 130 ++++ > > doc/guides/prog_guide/graph_lib.rst | 71 ++- > > doc/guides/sample_app_ug/l3_forward_graph.rst | 16 + > > examples/l3fwd-graph/main.c | 230 +++++-- > > lib/graph/graph.c | 161 +++++ > > lib/graph/graph_debug.c | 6 + > > lib/graph/graph_populate.c | 1 + > > lib/graph/graph_private.h | 90 +++ > > lib/graph/graph_stats.c | 76 ++- > > lib/graph/meson.build | 9 +- > > lib/graph/node.c | 27 +- > > lib/graph/rte_graph.h | 65 ++ > > lib/graph/rte_graph_model_mcore_dispatch.c | 191 ++++++ > > lib/graph/rte_graph_model_mcore_dispatch.h | 134 ++++ > > lib/graph/rte_graph_model_rtc.h | 46 ++ > > lib/graph/rte_graph_worker.c | 39 ++ > > lib/graph/rte_graph_worker.h | 503 +-------------- > > lib/graph/rte_graph_worker_common.h | 598 ++++++++++++++++++ > > lib/graph/version.map | 11 + > > 20 files changed, 1839 insertions(+), 568 deletions(-) create mode > > 100644 lib/graph/rte_graph_model_mcore_dispatch.c > > create mode 100644 lib/graph/rte_graph_model_mcore_dispatch.h > > create mode 100644 lib/graph/rte_graph_model_rtc.h create mode > > 100644 lib/graph/rte_graph_worker.c create mode 100644 > > lib/graph/rte_graph_worker_common.h > > > > -- > > 2.37.2 > >
