On Fri, May 21, 2021 at 2:14 PM Thomas Huth <th...@redhat.com> wrote: > > On 21/05/2021 16.29, Peter Maydell wrote: > > On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé <f4...@amsat.org> > > wrote: > >> If you think these tests belong to tests/tcg/, I am OK to put > >> them they, but I don't think adding the Avocado buildsys > >> machinery to the already-complex tests/tcg/ Makefiles is going > >> to help us... > > > > This does raise the question of what we're actually trying > > to distinguish. It seems to me somewhat that what tests/acceptance/ > > actually contains that makes it interestingly different from other > > tests/ stuff is that it's specifically "tests using the Avocado > > framework". On that theory we might name it tests/avocado/. > > I think there are two aspects: > > 1) These tests are using the avocado framework > > 2) These tests are downloading other stuff from the internet (unlike the > other tests that we have) >
After Peter's reply, I noticed QEMU does not organize tests under the tests folder by software engineering test category but by the mechanism/machinery the tests run on. This makes me think that we may need to handle the folders name and the CI jobs name differently: 1 - Change the current "test/acceptance" folder name to "test/(avocado or avoqado)." Change the "make check-acceptance" to "make check-validation," and the GitLab CI job names to "validation," meaning that, in a promising future, other tests running on a different framework and acting like validation tests would run in the same make command and same GitLab CI job. 2 - Change the current "test/acceptance" folder name to "test/(avocado or avoqado)." Change the "make check-acceptance" to "make check-(avocado or avoqaco)" and the GitLab CI job names to "(avocado or avoqado)," meaning that, in a promising future, we can categorize validation jobs inside the CI and run each of the different validation tests supported by a framework on its own GitLab CI job. Personally, I prefer option 2 as it gives more flexibility to decide how to set a GitLab CI job or run it when testing locally. > > Or we could just leave it as it is -- is the current naming > > actually confusing anybody? :-) > > Yes, I think "acceptance" is rather confusing. So far they haven't been part > of your PR acceptance tests (well, now they are part of the gitlab-CI, > though), and it's also not about tests that have been set up by customers, > which is what you normally think of when hearing "acceptance tests". So a > different name would be adequate. > > I think I'd vote for either "avocado", "avoqado" or "validation". > Even laughing every time I read "avoqado" (and thanks for that), I liked the idea as there is supplementary code added inside "tests/acceptance/avocado_qemu" to support the tests, meaning they are not "pure" avocado. > Thomas >
