linguini1 opened a new pull request, #3260:
URL: https://github.com/apache/nuttx-apps/pull/3260
## Summary
This PR introduces 'nest', the NuttX unit test collection. It is a
collection of
unit test suites built on top of the already integrated Unity test framework.
Running the application runs all of the enabled test cases with
pass/fail/skip
results summarized in the shell.
Partially solves #3258, but OS test could probably use some tweaking to get
the console output more readable.
Documentation for this application is coming soon, but I wanted to get
feedback on the initial implementation first.
## Impact
Ideally, this will slowly phase out OS test and replace it with more
structured
unit tests that have:
- Easily extendable code (simple to add new test cases with Unity)
- Configurable test cases (only need to run a small subset of tests
depending on
your change)
- Easily readable test summaries in output
- Well-documented test-cases (docstrings above test case functions,
descriptions
of suites on NuttX docs)
Since OS test is a favourite for checking regressions, hopefully nest can be
used to easily target subsystems to check for regressions and create a very
concise summary of the test results for reviewers.
## Testing
I tested compilation on a Linux machine. All of the initial tests I've
written
as a starting point for nest were tested using `sim:nsh`. Only one test fails
(for `/dev/zero`) and that is because I believe I actually discovered a bug
in
the `/dev/zero` implementation where its behaviour does not match Linux. I
will
patch/discuss this later.
Here is the summary of the test results on sim that is produced by running
`nest`:
```console
nsh> nest
nest_main.c:151:collections_cbuf__init_local:PASS
nest_main.c:152:collections_cbuf__init_malloc:PASS
nest_main.c:153:collections_cbuf__empty_local:PASS
nest_main.c:154:collections_cbuf__empty_malloc:PASS
nest_main.c:119:collections_hmap__create:PASS
nest_main.c:120:collections_hmap__create_r:PASS
nest_main.c:121:collections_hmap__create_zero:PASS
nest_main.c:122:collections_hmap__create_r_zero:PASS
nest_main.c:133:collections_list__empty_after_init:PASS
nest_main.c:134:collections_list__zerolen_after_init:PASS
nest_main.c:135:collections_list__notail_after_init:PASS
nest_main.c:136:collections_list__nohead_after_init:PASS
nest_main.c:137:collections_list__static_null_init:PASS
nest_main.c:238:devices_devascii__open_rdonly:PASS
nest_main.c:239:devices_devascii__open_rdwr:PASS
nest_main.c:240:devices_devascii__open_wronly:PASS
nest_main.c:242:devices_devascii__readzero:PASS
nest_main.c:243:devices_devascii__readline:PASS
nest_main.c:244:devices_devascii__readline_twice:PASS
nest_main.c:246:devices_devascii__writezero:PASS
nest_main.c:247:devices_devascii__writelarge:PASS
nest_main.c:155:devices_devconsole__open_rdonly:PASS
nest_main.c:156:devices_devconsole__open_rdwr:PASS
nest_main.c:157:devices_devconsole__open_wronly:PASS
nest_main.c:159:devices_devconsole__readzero:PASS
nest_main.c:160:devices_devconsole__writezero:PASS
nest_main.c:241:devices_devnull__open_rdonly:PASS
nest_main.c:242:devices_devnull__open_rdwr:PASS
nest_main.c:243:devices_devnull__open_wronly:PASS
nest_main.c:245:devices_devnull__readzero:PASS
nest_main.c:246:devices_devnull__readlarge:PASS
nest_main.c:248:devices_devnull__writezero:PASS
nest_main.c:249:devices_devnull__writelarge:PASS
nest_main.c:251:devices_devnull__wrrd:PASS
nest_main.c:252:devices_devurandom__open_rdonly:PASS
nest_main.c:253:devices_devurandom__open_rdwr:PASS
nest_main.c:254:devices_devurandom__open_wronly:PASS
nest_main.c:256:devices_devurandom__readzero:PASS
nest_main.c:257:devices_devurandom__writezero:PASS
nest_main.c:259:devices_devurandom__readlarge:PASS
nest_main.c:260:devices_devurandom__uniform:PASS
nest_main.c:256:devices_devzero__open_rdonly:PASS
nest_main.c:257:devices_devzero__open_rdwr:PASS
nest_main.c:258:devices_devzero__open_wronly:PASS
nest_main.c:260:devices_devzero__readzero:PASS
nest_main.c:138:devices_devzero__readlarge:FAIL: Expected 64 Was 0
nest_main.c:263:devices_devzero__writezero:PASS
nest_main.c:264:devices_devzero__writelarge:PASS
nest_main.c:266:devices_devzero__wrrd:PASS
-----------------------
49 Tests 1 Failures 0 Ignored
FAIL
```
I have also verified that enabling/disabling tests in Kconfig results in them
not being run. Disabling dependencies (i.e. setting `DEV_URANDOM=n` when
`TESTING_NEST_DEVICES_DEVURANDOM=y`) also causes the test cases dependent on
them to not be run (Kconfig automatically sets
`TESTING_NEST_DEVICES_DEVURANDOM=n`).
I am open to feedback if it would be preferred to replace tests missing
dependencies with a `TEST_IGNORE_MESSAGE("missing dependencies")` instead of
just excluding them from the build. I have opted for the latter to reduce
code
sizes and catch missing dependencies at compilation instead of runtime. It
also
uses Kconfig's existing features instead of the test-writer having to add
more
`#ifdefs`.
## Other Limitations
This is meant to be a unit test collection. Some things in OS test are not
necessarily "unit tests", like test cases involving multi-threading and
forcing
context switches, etc. I am still playing around with what can be done with
Unity; it may be possible to twist it into working with some multi-threaded
test
cases. That being said, this is not intended as a direct replacement of OS
test.
There is a reason that the test cases included in this PR are very easily
unit-testable subsystems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
