From: Alexis Lothoré <alexis.loth...@bootlin.com> This v2 does not contain any change in patches content, it only sets the From: field correctly. Sorry for the noise.
This patch serie is a proposal linked to discussion initiated here: https://lists.yoctoproject.org/g/automated-testing/topic/96652823#1219 After integration of some improvements on regression reporting, it has been observed that the regression report of version 4.2_M2 is way too big. When checking it, it appears that a big part of the report is composed of "missing tests" (regression detected because test status changed from "PASS" to "None"). It is mostly due to oeselftest results, since oeselftest is run multiple time for a single build, but not with the same parameters (so not the same tests "sets"), so those test sets are not comparable. The proposed serie introduce OSELFTEST_METADATA appended to tests results when the TEST_TYPE is "oeselftest". An oeselftest result with those metadata looks like this: [...] "configuration": { "HOST_DISTRO": "fedora-36", "HOST_NAME": "fedora36-ty-3", "LAYERS": { [...] }, "MACHINE": "qemux86", "STARTTIME": "20230126235248", "TESTSERIES": "qemux86", "TEST_TYPE": "oeselftest", "OESELFTEST_METADATA": { "run_all_tests": true, "run_tests": null, "skips": null, "machine": null, "select_tags": ["toolchain-user", "toolchain-system"], "exclude_tags": null } } [...] Additionally, the serie now makes resulttool look at a METADATA_MATCH_TABLE, which tells that when compared test results have a specific TEST_TYPE, it should look for some specific metadata to know if tests can be compared or not. It will then remove all the false positive in regression reports due to tests present in base results but not found in target results because of skipped tests/excluded tags * this serie prioritize retro-compatibility: if the base test is older (ie: it does not have the needed metadata), it will consider tests as "comparable" * additionally to tests added in oeqa test cases, some "best effort" manual testing has been done, with the following cases: - run a basic test (e.g: `oeselftest -r tinfoils`), collect test result, break test, collect result, ensure tests are compared. Change oeselftest parameters, ensure tests are not compared - collect base and target tests results from 4.2_M2 regression report, manually add new metadata to some tests, replay regression report, ensure that regressions are kept or discarded depending on the metadata Alexis Lothoré (4): scripts/oe-selftest: append metadata to tests results oeqa/selftest/resulttooltests: fix minor typo scripts/resulttool/regression: add metadata filtering for oeselftest oeqa/selftest/resulttool: add test for metadata filtering on regression .../oeqa/selftest/cases/resulttooltests.py | 123 +++++++++++++++++- meta/lib/oeqa/selftest/context.py | 15 ++- scripts/lib/resulttool/regression.py | 34 +++++ 3 files changed, 170 insertions(+), 2 deletions(-) -- 2.39.1
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#177161): https://lists.openembedded.org/g/openembedded-core/message/177161 Mute This Topic: https://lists.openembedded.org/mt/96964070/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
