On Thu, May 20, 2021 at 5:28 PM Philippe Mathieu-Daudé <phi...@redhat.com> wrote: > > On 5/20/21 9:53 PM, Willian Rampazzo wrote: > > Conceptually speaking, acceptance tests "are a series of specific tests > > conducted by the customer in an attempt to uncover product errors before > > accepting the software from the developer. Conducted by the end-user rather > > than software engineers, acceptance testing can range from an informal > > “test drive” to a planned and systematically executed series of scripted > > tests" [1]. Every time Pressman refers to the term "acceptance testing," he > > also refers to user's agreement in the final state of an implemented > > feature. > > Today, QEMU is not implementing user acceptance tests as described by > > Pressman. > > > > There are other three possible terms we could use to describe what is > > currently > > QEMU "acceptance" tests: > > > > 1 - Integration tests: > > - "Integration testing is a systematic technique for constructing the > > software architecture while at the same time conducting tests to > > uncover errors associated with interfacing. The objective is to > > take > > unit-tested components and build a program structure that has been > > dictated by design." [2] > > * Note: Sommerville does not have a clear definition of integration > > testing. He refers to incremental integration of components inside > > the system testing (see [3]). > > > > 2 - Validation tests: > > - "Validation testing begins at the culmination of integration > > testing, > > when individual components have been exercised, the software is > > completely assembled as a package, and interfacing errors have been > > uncovered and corrected. At the validation or system level, the > > distinction between different software categories disappears. > > Testing > > focuses on user-visible actions and user-recognizable output from > > the > > system." [4] > > - "where you expect the system to perform correctly using a set of > > test > > cases that reflect the system’s expected use." [5] > > * Note: the definition of "validation testing" from Sommerville > > reflects > > the same definition found around the Internet, as one of the > > processes > > inside the "Verification & Validation (V&V)." In this concept, > > validation testing is a high-level definition that covers unit > > testing, > > functional testing, integration testing, system testing, and > > acceptance > > testing. > > > > 3 - System tests: > > - "verifies that all elements mesh properly and that overall system > > function and performance is achieved." [6] > > - "involves integrating components to create a version of the system > > and > > then testing the integrated system. System testing checks that > > components are compatible, interact correctly, and transfer the > > right > > data at the right time across their interfaces." [7] > > > > The tests implemented inside the QEMU "acceptance" directory depend on the > > software completely assembled and, sometimes, on other elements, like > > operating > > system images. In this case, the proposal here is to rename the current > > "acceptance" directory to "system." > > Are user-mode tests using Avocado also system tests? > https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html
Yes, they are considered system tests because in software engineering, system testing means your software built and tested using external test artifacts, just like your example. > > > [1] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > > Practitioner’s Approach. p. 430. > > [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > > Practitioner’s Approach. Software Engineering, p. 398. > > [3] Sommerville, Ian (2016). Software Engineering. p. 240-242. > > [4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > > Practitioner’s Approach. Software Engineering, p. 407. > > [5] Sommerville, Ian (2016). Software Engineering. p. 227. > > [6] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > > Practitioner’s Approach. Software Engineering, p. 377. > > [7] Sommerville, Ian (2016). Software Engineering. p. 240. > > > > Signed-off-by: Willian Rampazzo <willi...@redhat.com> > > --- > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > > index f718b61fa7..c5de3c9fd5 100644 > > --- a/.gitlab-ci.yml > > +++ b/.gitlab-ci.yml > > @@ -52,7 +52,7 @@ include: > > # Avoid recompiling by hiding ninja with NINJA=":" > > - make NINJA=":" $MAKE_CHECK_ARGS > > > > -.acceptance_template: &acceptance_definition > > +.system_template: &system_definition > > .system_test_template: &system_test_definition ? Agreed, keeps the consistency. > > > diff --git a/tests/Makefile.include b/tests/Makefile.include > > index 8f220e15d1..c580292bb5 100644 > > --- a/tests/Makefile.include > > +++ b/tests/Makefile.include > > @@ -16,7 +16,7 @@ ifneq ($(filter $(all-check-targets), check-softfloat),) > > @echo " $(MAKE) check-tcg Run TCG tests" > > @echo " $(MAKE) check-softfloat Run FPU emulation tests" > > endif > > - @echo " $(MAKE) check-acceptance Run all acceptance (functional) > > tests" > > + @echo " $(MAKE) check-system Run all system tests" >
