On Fri, Apr 8, 2022 at 3:17 AM <ohily...@iol.unh.edu> wrote: > > From: Owen Hilyard <ohily...@iol.unh.edu> > > Currently, DTS uses Testpmd for most of its testing. This has been > successful in reducing the need to create more test apps, but it has a few > drawbacks. First, if some part of DPDK is not exposed via Testpmd or one of > the example applications, for the purposes of DTS it is not testable. This is > a situation I’d like to avoid. However, adding new functionality to Testpmd > is labor-intensive. Testpmd currently uses a hand-written LL(1) parser > (https://en.wikipedia.org/wiki/LL_parser) to parse command line options. This > makes adding new functionality difficult since the parser is stored as a > series of several thousand line long lookup tables. To look at it another > way, 64% of the 52238 lines in Testpmd are related to command line input in > some way. The command line interface of testpmd also presents several > challenges for the underlying implementation, since it requires that > everything a user might want to reference is identified via something that is > reasonable to ask a user to type. As of right now, this is handled via either > strings or integers. This can be handled by creating a global registry for > objects, but it is still extra work that I think can be avoided. In addition, > this leads to more places where things can go wrong. > > This is what DTS running a single command in testpmd looks like right now: > https://drive.google.com/file/d/1hvTcjfVdh8-I3CUNoq6bx82EuNQSK6qW/view?usp=sharing > > This approach has a number of disadvantages. First, it requires > assembling all commands as strings inside of the test suite and sending them > through a full round trip of SSH. This means that any non-trivial command, > such as creating an RTE flow, will involve a lot of string templating. This > normally wouldn’t be a big issue, except that some of the test suites are > designed to hundreds of commands over the course of a test, paying the cost > of an SSH round trip for each. Once Testpmd has the commands, it will then > call the appropriate functions inside of DPDK, and then print out all of the > state to standard out. All of this is sent back to DTS, where the author of > the test case then needs to handle all possible outputs of Trex, often by > either declaring the presence of a single word or short phrase in the output > as meaning success or failure. In my opinion, this is something that is > perfectly fine for humans to interact with, but it causes a lot of issues > with automation due to its inherent inflexibility and the less-than-ideal > methods of information transfer. This is why I am proposing the creation of > an automation-oriented pmd, with a focus on exposing as much. > > https://drive.google.com/file/d/1wj4-RnFPVERCzM8b68VJswAOEI9cg-X8/view?usp=sharing > > That diagram is a high-level overview of the design, which explicitly > excludes implementation details. However, it already has some benefits. > First, making DPDK do something is a normal method call, instead of needing > to format things into a string. This provides a much better interface for > people working in both DTS and DPDK. Second, the ability to return structured > data means that there won’t be parsers on both sides of communication > anymore. Structured data also allows much more verbosity, since it is no > longer an interface designed for humans. If a test case author needs to > return the bytes of every received packet back to DTS for comparison with the > expected value, they can. If you need to return a pointer for DTS to use > later, that becomes reasonable. Simply moving to shuffling structured data > around and using RPC already provides a lot of benefits. > The next obvious question would be what to use for the > implementation. The initial attempt was made using Python on both sides and > the standard library xmlrpc module. The RPC aspect of this approach worked > very well, with the ability to send arbitrary python objects back and forth > between DTS and app. However, having Python interacting with DPDK has a few > issues. First, DPDK is generally very multi-threaded and the most common > implementation of Python, CPython, does not have concurrency. It has > something known as the global interpretr lock, which is a global mutex. This > makes it very difficult to interact with blocking, multi-threaded code. The > other issue is that I was not able to find a binding generator that I feel > would be sufficient for DPDK. Many generators assumed sizeof(int) == 4 or had > other portability issues such as assuming GCC or Clang as a C compiler. > Others focused on some subset of C, meaning they would throw errors on > alignment annotations. > Given this, I decided to look for cross-language RPC libraries. Although > libraries exist for performing xmlrpc in C, they generally appeared quite > difficult to use and required a lot of manual work. The next best option was > gRPC. gRPC allows using a simple language, protobuf, with a language > extension for rpc. It provides code generation to make it easy to use > multiple languages together, since it was developed to make polyglot > microservice interaction easier. The only drawback is that it considers C++ > good enough for C support. In this case, I was able to easily integrate DPDK > with C++, so that isn’t much of a concern. I used C++17 in the attached > patches, but the minimum requirements are C++11. If there is concern about > modern C++ causing too much mental overhead, a “C with classes” subset of C++ > could easily be used. I also added an on-by-default option to use a C++ > compiler, allowing anyone who does not have a C++ compiler available to them > to turn off everything that uses C++. This disables the application written > for this RFC. > One of the major benefits of gRPC is the asynchronous API. This allows > streaming data on both sides of an RPC call. This allows streaming logs back > to DTS, streaming large amounts of data from low-memory systems back to DTS > for processing, and would allow DTS to easily feed DPDK data, ending the test > quickly on a failure. Currently, due to the overhead of sending data to > Testpmd, it is common to just send all of the commands over and run > everything since that will be much faster when the test passes, but it can > cost a lot of time in the event of a failure. There are also optional > security features for requiring authentication before allowing code > execution. I think that a discussion on whether using them for DTS is > necessary is warranted, although I personally think that it’s not worth the > effort given the type of environment this sample application is expected to > run in. > For this RFC, I ported test-acl because it was mostly self-contained and > was something I could run on my laptop. It should be fairly easy to see how > you would expand this proof of concept to cover more of DPDK, and I think > that most of the functions currently used in testpmd could be ported over to > this approach, saving a lot of development time. However, I would like to see > some more interest before I take on a task like that. This will require a lot > of work on the DTS side to implement, but it will make it much easier to add > new features to DTS.
Thanks, Owen for POC. In my view, May using this scheme is probably over-engineered. The reason for thinking so is -Now that, Test code is also part of DPDK, Exposing as services may not be required. -Now in DPDK, we have two types of existing test cases to verify the API -- Noninteractive - These test cases, can simply run over ssh with bash invocation and return the test from a remote PC, -- Interactive - Testpmd one, I believe, Feeding stdin programmatically would suffice to test all the combinations. -We need to add all test cases in this model and we need to maintain two sets of programs.(Traditional tests and gRPC model-based tests). Just my 2c. > > Owen Hilyard (4): > app/test-pmd-api: Add C++ Compiler > app/test-pmd-api: Add POC with gRPC deps > app/test-pmd-api: Add protobuf file > app/test-pmd-api: Implementation files for the API > > app/meson.build | 17 + > app/test-pmd-api/api.proto | 12 + > app/test-pmd-api/api_impl.cc | 1160 ++++++++++++++++++++++++++++++++++ > app/test-pmd-api/api_impl.h | 10 + > app/test-pmd-api/main.c | 11 + > app/test-pmd-api/meson.build | 96 +++ > meson.build | 3 + > meson_options.txt | 2 + > 8 files changed, 1311 insertions(+) > create mode 100644 app/test-pmd-api/api.proto > create mode 100644 app/test-pmd-api/api_impl.cc > create mode 100644 app/test-pmd-api/api_impl.h > create mode 100644 app/test-pmd-api/main.c > create mode 100644 app/test-pmd-api/meson.build > > -- > 2.30.2 >
