On Wed, Jan 27, 2021 at 10:16:18AM -0500, Aaron Conole wrote: > The DPDK testing infrastructure includes a comprehensive set of > libraries, utilities, and CI integrations for developers to test > their code changes. This isn't well documented, however. > > Document the basics for adding a test suite to the infrastructure > and enabling that test suite for continuous integration platforms > so that newer developers can understand how to develop test suites > and test cases. > > Signed-off-by: Aaron Conole <acon...@redhat.com>
Thanks for starting this Aaron, this doc is well needed. Couple of comments below. /Bruce > --- > Submitting as RFC as I'm not sure if this should include making > changes to the github actions or travis yml, or the linux-build > ci script. > > doc/guides/contributing/index.rst | 1 + > doc/guides/contributing/testing.rst | 200 ++++++++++++++++++++++++++++ > 2 files changed, 201 insertions(+) > create mode 100644 doc/guides/contributing/testing.rst > <snip> > +The second form is useful for a scripting environment, and is used by > +the DPDK meson build system. This mode is invoked by assigning a > +specific test suite name to the environment variable `DPDK_TEST` > +before invoking the `dpdk-test` command, such as:: > + > + $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --no-huge > + EAL: Detected 4 lcore(s) > + EAL: Detected 1 NUMA nodes > + EAL: Static memory layout is selected, amount of reserved memory can be > adjusted with -m or --socket-mem > + EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket > + EAL: Selected IOVA mode 'VA' > + EAL: Probing VFIO support... > + EAL: PCI device 0000:00:1f.6 on NUMA socket -1 > + EAL: Invalid NUMA socket, default to 0 > + EAL: probe driver: 8086:15d7 net_e1000_em > + APP: HPET is not enabled, using TSC as default timer > + RTE>>version_autotest > + Version string: 'DPDK 20.02.0-rc0' > + Test OK > + RTE>>$ > + > +The above shows running a specific test case. On success, the return > +code will be '0', otherwise it will be set to some error value (such > +as '255'). > + This reminds me that I have patch almost ready to submit to extend this support to allow passing the test name - or even multiple test names - on the commandline, not just through the environment. Will upstream it shortly, I hope. I think command-line is more usable for folks than the environment. Thoughts? > + <snip> > + > +Designing a test > +---------------- > + > +Test cases have multiple ways of indicating an error has occurred, > +in order to reflect failure state back to the runner. Using the > +various methods of indicating errors can assist in not only validating > +the requisite functionality is working, but also to help debug when > +a change in environment or code has caused things to go wrong. > + > +The first way to indicate a generic error is by returning a test > +result failure, using the *TEST_FAILED* error code. This is the most > +basic way of indicating that an error has occurred in a test routine. > +It isn't very informative to the user, so it should really be used in > +cases where the test has catastrophically failed. > + > +The preferred method of indicating an error is via the > +`RTE_TEST_ASSERT` family of macros, which will immediately return > +*TEST_FAILED* error condition, but will also log details about the > +failure. The basic form is: > + > +.. code-block:: c > + > + RTE_TEST_ASSERT(cond, msg, ...) > + > +In the above macro, *cond* is the condition to evaluate to **true**. > +Any generic condition can go here. The *msg* parameter will be a > +message to display if *cond* evaluates to **false**. Some specialized > +macros already exist. See `lib/librte_eal/include/rte_test.h` for > +a list of pre-build test assertions. > + Maybe also mention TEST_SKIPPED return value.
