What is the long term plan for package automated testing?
https://fedoraproject.org/wiki/Category:Test_Cases
E.g. 18264 packages * 20 tests run continuously by humans is a lot work. The
first question I expect many package maintainers to raise is that its a much
better investment to maintain automated tests instead. As a user, I only trust
that tests were run if I can see execution logs.
I agree that upstream + package maintainers should be supplying per-package
tests. and IMHO the source package is the most logical place to store and
generate them from.
SRPMs should be providing either:
- include a standardized test section similar to DEB autopkgtest ("in-build"
tests)
- OR building produces a "-tests" package which puts a bunch of simple
returncode test executables under somewhere standard. Example:
/usr/lib/testing/<source pkg name>/bin/{test_foo,test_abc} and group them in
plaintext lists /usr/lib/testing/<source pkg
name>/testgroups/{quick,build,qa_intensive,smoke} etc. (Executing the tests
could emit coverage information to some known location. (gcov, pytest etc))
The seperate "-tests" RPM has a few advantages over in-build tests, including:
- Tests the final product: the built binary RPM installed on a working system
(not the in-build environment)
- Different test groups for different phases of package release process
- Users can easily run test X seperately for troubleshooting
- Makes the tests easily usable during new build contexts (rpm ostree,
container build testing etc)
- System-wide integration tests could simply be stored in their own package.
--
test mailing list
test@lists.fedoraproject.org
To unsubscribe:
https://lists.fedoraproject.org/admin/lists/test@lists.fedoraproject.org
