IMO it should be added to the GLEP that then TEST_SUITE_PRESENT should be true (exists).
W dniu 19.02.2023 o 18:32, Michał Górny pisze:
Signed-off-by: Michał Górny <mgo...@gentoo.org> --- glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 glep-9999.ebuild diff --git a/glep-9999.ebuild b/glep-9999.ebuild new file mode 100644 index 0000000..9ee18ca --- /dev/null +++ b/glep-9999.ebuild @@ -0,0 +1,132 @@ +--- +GLEP: 9999 +Title: TEST_SUITE_PRESENT variable +Author: Michał Górny <mgo...@gentoo.org> +Type: Standards Track +Status: Draft +Version: 1 +Created: 2023-02-19 +Last-Modified: 2023-02-19 +Post-History: 2023-02-19 +Content-Type: text/x-rst +--- + + +Abstract +======== + +A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether +the package features a test suite. It can be set either by the ebuild, +the eclass or the default ``src_test`` implementation, and afterwards +included in the package manager logs. This can aid in analyzing +the results of automated package testing. + + +Motivation +========== + +The deployment of new Python targets in Gentoo currently involves +testing of a large number of Gentoo packages against the said target. +This is currently done manually for the most part. It can be +particularly time consuming if multiple individuals repeatedly test +the same package only to determine that it remains incompatible with +the new interpreter. + +The Python team wanted to explore the use of automation to aid this +testing. Unfortunately, this faces a major problem: for the vast +of majority of packages, the incompatibilities with new Python versions +do not exhibit during the installation and can only be detected through +running the test suite. The results of automated testing are therefore +only meaningful if the package features a test phase. + +For packages using ``distutils-r1`` eclass, the presence of test suite +can usually be easily determined through grepping for +``distutils_enable_tests`` call or an explicit ``python_test()`` +function. Even then, it seems sensible to work towards a more generic +approach to tell whether a package had a test suite or not, +and therefore whether a particular successful automated testing result +means that the package actually passed tests or only confirmed that +the Python files were copied successfully. + +An explicit indication whether a test suite was present can be presented +by the package manager as part of logs, along with the result of running +the test phase. Afterwards, these logs can be used to determine which +packages were actually tested. + + +Specification +============= + +A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set +by a ``src_test()`` implementation to indicate whether the package +featured a test suite. It can take three values: + +- ``yes`` indicating that a test suite was run +- ``indeterminate`` indicating that it was not possible to clearly + determine whether the test suite was present or not (this could be + a case e.g. when a generic test command is run and it does not + indicate whether any tests were found) +- ``no`` indicating that no test suite was run + +This variable *should* be set by eclasses defining the ``src_test()`` +phase. If the package in question is using ``src_test()`` defined +by an eclass that does not declare it explicitly, the PM must assume +``indeterminate``. + +The variable *may* be set by an ebuild defining the ``src_test()`` +phase. If the ebuild does not define it explicitly, the PM must assume +``yes``. + +The default ``src_test()`` implementation as defined by the PMS sets +the value to ``indeterminate`` if it runs a ``check`` or ``test`` +target, and to ``no`` if neither of the targets is found. + + +Rationale +========= + +The use of ternary flag makes it possible to clearly represent all three +possible outcomes while navigating the defaults defined in the GLEP. +The flag is set in ``src_test()``, so that runtime conditions (such +as the results obtained from the actual test runner) can be used to +determine the actual value. + +The defaults were defined based on the following assumptions: + +1. The presence of ``check`` target is common in autotools projects but + it does not guarantee that the target actually does anything, let + alone run a proper test suite. However, the lack of any test target + clearly indicates that no tests were run. + +2. Eclass ``src_test`` implementations can be very generic and succeed + without actually performing any testing. It is therefore reasonable + to default to ``indeterminate`` result when they are used, + and recommend them to explicitly override the variable. + +3. Explicit ``src_test`` declared in ebuild can generally be assumed + to actually run tests, with the exception of declaring the function + to prevent ``default_src_test`` from running. It therefore makes + sense to default to ``yes`` but allow ebuilds to override the value + in the latter case. + + +Backwards Compatibility +======================= + +This GLEP is entirely optional. The package managers not implementing +it will treat the variable as a regular bash variable set by the test +phase and ignore it. + + +Reference Implementation +======================== + +TODO + + +Copyright +========= + +This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 +International License. To view a copy of this license, visit +https://creativecommons.org/licenses/by-sa/4.0/.
-- Have a great day! ~ Maciej XGQT Barć x...@gentoo.org Gentoo Linux developer (dotnet, emacs, math, ml, nim, scheme, sci) https://wiki.gentoo.org/wiki/User:Xgqt 9B0A 4C5D 02A3 B43C 9D6F D6B1 14D7 4A1F 43A6 AC3C
OpenPGP_0x14D74A1F43A6AC3C.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature
