Hello world :-) As there are some works ongoing to improve NuttX testing (nxdart, ntfc, nuts) we are coming back to to discussion about existing (and possibly new) board configurations that would help in testing.
Initial discussion [1] proposed four levels of testing: base, extended, specific, custom. Base should cover all boards and constitute mandatory implementation. Extended tests can be optional and contain stresstests and benchmarks. So far lots of boards had separate nsh, ostest, coremark, getprime, configurations, that would fit in as the candidates for base and extended tests. Looks like all boards hould have base and extended tests defined. But we did not define them back then. After recent release testing I noted down missing ostest for some boards [2] to keep things aligned to what we already have. Raiden raised concern that this makes NuttX more complex and came out with idea to bundle all testing utilities where possible on bigger boards into one configuration - because that would simplify citesting, require only one build, and allow different runtime tests to be performed with a single configuration. I like the idea myself.. with caution that not all boards would fit such a bundle :-) For now configurations are a bit incoherent - for instance some :nsh configurations only contain bare nsh, with different built-in commands, some additionally contain ostest, getprime, etc, while other boards have them as separate configurations. The main idea is to clean up existing configurations that would keep them coherent, predictable, and ci/automation friendly. There are several things to discuss to get things coherent and documented: 1. How to organize test configurations [1]? Is the 4 tier granularity okay? Do we want something else? 2. What should we consider as mandatory :test-base configuration? What should it contain? nsh? free? ostest? coremark? getprime? What exact applications and script? 3. What should we consider as optional :test-ext configuration? What should it contain? coremark? getprime? What exact application and script? 4. Do we agree to have one big test configuration for big boards where they fit? 5. If we agree on (4) then would is be okay to remove existing configurations with applications that would be part of the :test-base and :test-ext bundle (i.e. :ostest that would go to :test-base and :coremark that would go to :test-ext)? This would simplify CI with less configurations to build. 6. What happens if smaller board cannot fit all of the test-base applications? Should we still have separate configurations for example test-base-ostest / old ostest, test-ext-getprime / old getprime, etc? 7. What should be the name for the test bundle? Do we want to keep it under "nsh" or new "test-base"? Note from Raiden suggests that less configurations is better for CI/build/automation. I know this discussion may result in existing configurations change, thus may be breaking change, thus we may need a vote after details are set if necessary :-) If there is anything missing please add! Any hints welcome :-) Tomek [1] https://github.com/apache/nuttx/issues/15773 [2] https://github.com/apache/nuttx/issues/17721 -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
