cederom opened a new issue, #15773:
URL: https://github.com/apache/nuttx/issues/15773
### Description
* In order to provide generic approach and universal architecture we need to
define a standard set of test cases.
* There are two main categories of tests: Build and Runtime.
* Runtime tests can be: Internal (working as the NuttX binary application
"inside" the MCU, no external components required, even onboard), and External
(NuttX interaction with external world outside MCU takes place, i.e. sensors,
network, WIFI, BLE, GPIO, other interfaces and peripherals).
## Build Tests
* Mandatory: Build current `nuttx` and `nuttx-apps` master. This ensures
current HEAD quality.
* Optional(?): Build current `nuttx` with older `nuttx-apps` (i.e. release
package/tag or just N-steps back in repo history). This ensures backward
compatibility between `nuttx` and `nuttx-apps`.
* Mandatory: Cover each architecture (MCUs to be selected).
* Optional: Cover each possible arch/board.
* Others?
## Runtime Tests
### Runtime Tests: Base
* These tests are mandatory.
* Must be implemented and supported on every board under evaluation.
* Ensures quality of builds provided by Build Tests.
* Test Cases (to be defined):
* `nsh`.
* `uname -a`.
* `help`.
* `ostest`.
* `free`.
### Runtime Tests: Extended
* These tests are optional.
* Ensures performance validation (stability/improvement/degradation).
* Test Cases (to be defined):
* stress tests.
* benchmarks.
### Runtime Tests: Specific
* These tests are optional, they may be hardware specific (thus name).
* Ensures architecture / board / peripheral validation.
* If generic tests are used on different boards with no hardware support
then `UNAVAILABLE` should be returned rather than `FAILED`.
* Test Cases (to be defined):
* Ethernet.
* WIFI.
* Bluetooth.
* LoRa.
* LCD.
* Touchscreen.
* Joystick.
### Runtime Tests: Custom
* Vendors must be able to re-use our framework on-premises with custom
hardware to perform their own tests in a generic way.
* These tests can also reveal problems with our code (i.e. compatibility,
efficiency, etc)
Comments below are welcome :-)
### Verification
- [x] I have verified before submitting the report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@nuttx.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
