Do we have the enough resource to develop a full test suite from scratch? Before we get the resource commitment, the realistic step is integrate the test suite from other project. This isn't a bad idea because NuttX is a ANSI/POSIX compliant OS, there is many test suite to cover POSIX, C/C++ library. So what I suggest is that: 1.Prepare the golden config(evolute with test case) run on QEMU for each arch 2.Review apps folder to gather the application which is suitable to become test 3.Enhance the workflow to run the builtin basic test suite In step 2 and 3, some modification may require to achieve the automation run and check. Once the basic workflow setup, we can: 1.Integrate the test suite from other project 2.Develop our own test for the NuttX specific feature I am wondering it's possible to test on the real hardware if we don't get the donation from the commercial company.
Thanks Xiang On Wed, May 20, 2020 at 3:37 AM Gregory Nutt <spudan...@gmail.com> wrote: > > Before we select any tools, I think we should first enumerate some > general requirements of the automated test. Here are a few things that > occur to me. > > I think it should be pretty simple from the standpoint of features; > > * Execute a sequence of test cases, broken out as test stops. > * It must supports some kind of test scripting language. > * It must support some kinds of "plug-ins" or libraries that we > could use for custom board interfaces to reset and load code and to > get console output. > * That should be some extensible extensible interfaces to that we > can add more "plug-ins" in the future .. and somehow bound into the > test scripting language. > * Some way to generate test reports and identify regressions. > > What else? Python seems to be the most common scripting language for > this kind of application. >
