v2: - Added macro in place of testcase/testsuite loops. - Added more detail in the summary output. - Moved testcase counts to the testsuite structure. - Flattened testsuite structure to remove union. - Added patch for fix of blockcipher test return value. - Squashed release note into last patch.
The current crypto unit test framework is not granular enough to accurately track unit test results. This is caused by one testcase in a suite actually running multiple testcases, but only returning one result. The approach taken in this patchset allows a test suite have either a list of sub-testsuites, or a list of testcases as previously used. The unit test suite runner can then recursively iterate and run the sub-testsuites, until it reaches a suite with testcases, and it then runs each testcase as it had done previously. By allowing this further breakdown into sub-testsuites, a refactor of the crypto unit tests solves the issue of inaccurate reporting, as sub-testsuites can be used in place of the testcases that had multiple testcases hidden on a sub level. The blockcipher tests previously had these hidden testcases, but are now sub-testsuites that are dynamically created and added to a parent test suite, allowing for each testcase status to be reported directly to the runner. The cryptodev test suite is broken down into smaller suites that are used as sub-testsuites, which allows for more flexibility choosing which sub-testsuites should run for the current device autotest. The introduction of sub-testsuites also allows for more precise setup/teardown functions, rather than general ones loaded with conditions as was seen with the initial setup function used for all crypto testsuites. For example, when running the cryptodev_aesni_mb_autotest, the AESNI MB parent test suite has its own setup function to initialise the AESNI MB device. Various sub-testsuites are added to the parent test suite, such as some of the static suites that were once in the cryptodev_testsuite, and blockcipher suites. The unit test runner can then run the AESNI MB parent test suite, which in turn will run the sub-testsuites. Documentation will be added in a later version of the patchset, adding to the test document that isn't yet merged. [1] --- [1] https://patchwork.dpdk.org/project/dpdk/patch/20210309155757.615536-1-acon...@redhat.com/ Ciara Power (6): app/test: refactor of unit test suite runner test: introduce parent testsuite format test/crypto: refactor to use sub-testsuites test/crypto: move testsuite params to header file test/crypto: fix return value on test skipped test/crypto: dynamically build blockcipher suite app/test/test.c | 226 ++- app/test/test.h | 23 +- app/test/test_cryptodev.c | 1964 ++++++++++++++++++------ app/test/test_cryptodev.h | 20 + app/test/test_cryptodev_asym.c | 93 +- app/test/test_cryptodev_blockcipher.c | 127 +- app/test/test_cryptodev_blockcipher.h | 12 +- app/test/test_ipsec.c | 32 +- doc/guides/rel_notes/release_21_05.rst | 5 + 9 files changed, 1831 insertions(+), 671 deletions(-) -- 2.25.1
