On Sun, Feb 19, 2023 at 3:11 PM Maciej Barć <x...@gentoo.org> wrote: > > What if developer configured an ebuild in a way that it downloads the > test suite/files/data with USE=test? > > IMO it should be added to the GLEP that then TEST_SUITE_PRESENT should > be true (exists).
This is what I was afraid of with the name; it's not only about a testsuite being there or not, it's about trusting that: - There is a test suite. - It was run. - It passed, and passing is valuable. Just existing is not sufficient, because we cannot gain any information from an src_test that just downloaded but didn't run the suite. That is not valuable information. -A > > W dniu 19.02.2023 o 18:32, Michał Górny pisze: > > Signed-off-by: Michał Górny <mgo...@gentoo.org> > > --- > > glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 132 insertions(+) > > create mode 100644 glep-9999.ebuild > > > > diff --git a/glep-9999.ebuild b/glep-9999.ebuild > > new file mode 100644 > > index 0000000..9ee18ca > > --- /dev/null > > +++ b/glep-9999.ebuild > > @@ -0,0 +1,132 @@ > > +--- > > +GLEP: 9999 > > +Title: TEST_SUITE_PRESENT variable > > +Author: Michał Górny <mgo...@gentoo.org> > > +Type: Standards Track > > +Status: Draft > > +Version: 1 > > +Created: 2023-02-19 > > +Last-Modified: 2023-02-19 > > +Post-History: 2023-02-19 > > +Content-Type: text/x-rst > > +--- > > + > > + > > +Abstract > > +======== > > + > > +A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether > > +the package features a test suite. It can be set either by the ebuild, > > +the eclass or the default ``src_test`` implementation, and afterwards > > +included in the package manager logs. This can aid in analyzing > > +the results of automated package testing. > > + > > + > > +Motivation > > +========== > > + > > +The deployment of new Python targets in Gentoo currently involves > > +testing of a large number of Gentoo packages against the said target. > > +This is currently done manually for the most part. It can be > > +particularly time consuming if multiple individuals repeatedly test > > +the same package only to determine that it remains incompatible with > > +the new interpreter. > > + > > +The Python team wanted to explore the use of automation to aid this > > +testing. Unfortunately, this faces a major problem: for the vast > > +of majority of packages, the incompatibilities with new Python versions > > +do not exhibit during the installation and can only be detected through > > +running the test suite. The results of automated testing are therefore > > +only meaningful if the package features a test phase. > > + > > +For packages using ``distutils-r1`` eclass, the presence of test suite > > +can usually be easily determined through grepping for > > +``distutils_enable_tests`` call or an explicit ``python_test()`` > > +function. Even then, it seems sensible to work towards a more generic > > +approach to tell whether a package had a test suite or not, > > +and therefore whether a particular successful automated testing result > > +means that the package actually passed tests or only confirmed that > > +the Python files were copied successfully. > > + > > +An explicit indication whether a test suite was present can be presented > > +by the package manager as part of logs, along with the result of running > > +the test phase. Afterwards, these logs can be used to determine which > > +packages were actually tested. > > + > > + > > +Specification > > +============= > > + > > +A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set > > +by a ``src_test()`` implementation to indicate whether the package > > +featured a test suite. It can take three values: > > + > > +- ``yes`` indicating that a test suite was run > > +- ``indeterminate`` indicating that it was not possible to clearly > > + determine whether the test suite was present or not (this could be > > + a case e.g. when a generic test command is run and it does not > > + indicate whether any tests were found) > > +- ``no`` indicating that no test suite was run > > + > > +This variable *should* be set by eclasses defining the ``src_test()`` > > +phase. If the package in question is using ``src_test()`` defined > > +by an eclass that does not declare it explicitly, the PM must assume > > +``indeterminate``. > > + > > +The variable *may* be set by an ebuild defining the ``src_test()`` > > +phase. If the ebuild does not define it explicitly, the PM must assume > > +``yes``. > > + > > +The default ``src_test()`` implementation as defined by the PMS sets > > +the value to ``indeterminate`` if it runs a ``check`` or ``test`` > > +target, and to ``no`` if neither of the targets is found. > > + > > + > > +Rationale > > +========= > > + > > +The use of ternary flag makes it possible to clearly represent all three > > +possible outcomes while navigating the defaults defined in the GLEP. > > +The flag is set in ``src_test()``, so that runtime conditions (such > > +as the results obtained from the actual test runner) can be used to > > +determine the actual value. > > + > > +The defaults were defined based on the following assumptions: > > + > > +1. The presence of ``check`` target is common in autotools projects but > > + it does not guarantee that the target actually does anything, let > > + alone run a proper test suite. However, the lack of any test target > > + clearly indicates that no tests were run. > > + > > +2. Eclass ``src_test`` implementations can be very generic and succeed > > + without actually performing any testing. It is therefore reasonable > > + to default to ``indeterminate`` result when they are used, > > + and recommend them to explicitly override the variable. > > + > > +3. Explicit ``src_test`` declared in ebuild can generally be assumed > > + to actually run tests, with the exception of declaring the function > > + to prevent ``default_src_test`` from running. It therefore makes > > + sense to default to ``yes`` but allow ebuilds to override the value > > + in the latter case. > > + > > + > > +Backwards Compatibility > > +======================= > > + > > +This GLEP is entirely optional. The package managers not implementing > > +it will treat the variable as a regular bash variable set by the test > > +phase and ignore it. > > + > > + > > +Reference Implementation > > +======================== > > + > > +TODO > > + > > + > > +Copyright > > +========= > > + > > +This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 > > +International License. To view a copy of this license, visit > > +https://creativecommons.org/licenses/by-sa/4.0/. > > -- > Have a great day! > > ~ Maciej XGQT Barć > > x...@gentoo.org > Gentoo Linux developer > (dotnet, emacs, math, ml, nim, scheme, sci) > https://wiki.gentoo.org/wiki/User:Xgqt > 9B0A 4C5D 02A3 B43C 9D6F D6B1 14D7 4A1F 43A6 AC3C
