> > On Tue, 2021-12-14 at 14:12 +0000, Ronan Randles wrote: > > > This patchset introduces a Gen library for DPDK. This library provides an > > > easy > > > way to generate traffic in order to test software based network > > > components. > > > > > > This library enables the basic functionality required in the traffic > > > generator. > > > This includes: raw data setting, packet Tx and Rx, creation and > > > destruction of a > > > Gen instance and various types of data parsing. > > > This functionality is implemented in "lib/gen/rte_gen.c". IPv4 parsing > > > functionality is also added in "lib/net/rte_ip.c", this is then used in > > > the gen > > > library. > > > > > > A sample app is included in "examples/generator" which shows the use of > > > the > > gen > > > library in making a traffic generator. This can be used to generate > > > traffic by > > > running the dpdk-generator generator executable. This sample app supports > > > runtime stats reporting (/gen/stats) and line rate limiting > > > (/gen/mpps,<target traffic rate in mpps>) through telemetry.py. > > > > > > As more features are added to the gen library, the sample application will > > > become more powerful through the "/gen/packet" string parameter > > > (currently supports IP and Ether address setting). This will allow every > > > application to generate more complex traffic types in the future without > > > changing API. > > > > A good start, thanks. > > > > Testpmd can do txonly, rxonly, forwarding, but no rx+tx only :) I guess > > that's the key of a GEN. testpmd also has "--stats-period" to show > > statistics in non-interactive mode, very close to a GEN, but not > > enough. Testpmd can toggle a lot of settings like offloading, that's an > > advantage. Do you see an chance to make the lib part of testpmd? > > > > I tried to leverage Scapy syntax into testpmd [1][2]: to set fancy > > template - useful to test flows, or dump packet using different scapy > > detail level. unfotunately I don't have time to finish it. Parsing > > packet from string is boring, any consideration on this? > > Hi Steven, > > Yes I recall the patchset to extend DPDK with python/scapy itself, and indeed > this was my first approach too! > I build a proof-of-concept for enabling a > "run-python-script-inside-eth-null-PMD-rx-function" approach. > Although this worked (in a clunky way, using packet.__bytes__() in python > script..), the performance was > extremely low. An optimization was to pre-calculate e.g. 1 million packets, > and leave them all in memory, > which causes memory waste, and lack of performance due to memory-boundedness > in the CPU. > > There is a fundamental blocking issue here - the generation/creation of > packets is slow. Even updating a > single flow, or single parameter of the packet, could/would require > recalculation of checksums... so leads > to the C code needing to know the L2/L3/L4 offsets. Even then, the > re-calculation over 1-million packets > puts a lot of memory pressure - the packets cannot be re-used (from mempool > lcore cache) if they need > to be "unique" through pre-computation, resulting in cache-trashing. > > As a solution, moving to a native/C language based creation of a "base > packet" (template) with "modifiers" > seemed a good fit. It allows best-case performance (mempool cache) and lowest > compute per-packet. > The "modifiers" are "pay for what you need" approach to flows/updates/etc, > allowing user to choose > the right balance of traffic-complexity vs CPU-cost to generate it. > > All in all; handling string-parsing in C is not fun - but by paying that > price we can gain many other things...
Just to throw in an alternate thought: Instead of trying to re-implement scapy's abilities and face all related complexities (string parsings, etc.) - might be choose another way: allow user to write his own packet fillers via eBPF? I.E: user write and compile eBPF program that will fill l2/l3/l4 headers, set mbuf flags, etc. Then it will be loaded by packet-generator and executed for each outgoing packet. That way user will have full flexibility and extensibility in defining his outgoing traffic. Again from developer/maintainer point of view such packet-gen app will probably require much less effort. Though yes - user will need to do more work on his own - write actual packet filler code. > > I think that a "gen" engine for TestPMD could be interesting for certain > use-cases yes. As your links below, > the "expect" behaviour is a particularly nice way of working for testing! > There is an implementation for > net/null PMD to create packets with Gen library, but is not ready for > upstream so was not included in V2. > > For multi-flow latency testing, I'm not sure if TestPMD is the ideal tool, as > it does not prioritize RX/latency > at all costs: e.g lcores perform both RX and TX functionality. The dedicated > "example/generator" is a sample > application that provides the basis for a more latency/jitter focused > application, however there is much work > to do there to make it a production-ready tool. > > Note that at the moment, the status of the Gen work is as follows; > - Upstream discussion: > http://mails.dpdk.org/archives/dev/2022-January/232965.html > - V2 patch: > http://patches.dpdk.org/project/dpdk/cover/20220121103122.2926856-1-ronan.rand...@intel.com/ > > Regards, -Harry > > PS: Side note around Python/Scapy: its an *awesome* tool. I like it a lot, > but for DPDK levels of performance, > and in a latency/performance critical use-case, I'm not convinced it is the > right tool to use. > > > > [1]: https://github.com/steevenlee/dpdk/commits/upstream_scapy > > [2]: doc > > https://github.com/steevenlee/dpdk/blob/875b8407f769465837513d29a495af3cc > > 1a29765/doc/guides/howto/scapy.rst > > > > > > > > Harry van Haaren (6): > > > gen: add files for initial traffic generation library > > > gen: add basic Rx and Tx routines and tests > > > gen: add raw packet data API and tests > > > gen: add parsing infrastructure and Ether protocol > > > gen: add gen IP parsing > > > examples/generator: import code from basicfwd.c > > > > > > Ronan Randles (6): > > > net: add string to IPv4 parse function > > > net: add function to pretty print IPv4 > > > examples/generator: enable gen library for traffic gen > > > examples/generator: telemetry support > > > examples/generator: link status check added > > > examples/generator: line rate limiting > > > > > > app/test/meson.build | 4 + > > > app/test/test_gen.c | 184 +++++++++++ > > > app/test/test_net.c | 87 ++++++ > > > doc/api/doxy-api-index.md | 3 +- > > > doc/api/doxy-api.conf.in | 1 + > > > examples/generator/main.c | 483 ++++++++++++++++++++++++++++ > > > examples/generator/meson.build | 13 + > > > examples/meson.build | 1 + > > > lib/gen/meson.build | 6 + > > > lib/gen/rte_gen.c | 553 +++++++++++++++++++++++++++++++++ > > > lib/gen/rte_gen.h | 114 +++++++ > > > lib/gen/version.map | 10 + > > > lib/meson.build | 1 + > > > lib/net/meson.build | 1 + > > > lib/net/rte_ip.c | 58 ++++ > > > lib/net/rte_ip.h | 38 +++ > > > lib/net/version.map | 9 + > > > 17 files changed, 1565 insertions(+), 1 deletion(-) > > > create mode 100644 app/test/test_gen.c > > > create mode 100644 app/test/test_net.c > > > create mode 100644 examples/generator/main.c > > > create mode 100644 examples/generator/meson.build > > > create mode 100644 lib/gen/meson.build > > > create mode 100644 lib/gen/rte_gen.c > > > create mode 100644 lib/gen/rte_gen.h > > > create mode 100644 lib/gen/version.map > > > create mode 100644 lib/net/rte_ip.c > > >
