Hi! On Tue, 17 May 2016, Kent Fredric wrote:
> On 17 May 2016 at 19:37, Pallav Agarwal <pallavagarwa...@gmail.com> wrote: > > For normal users we wouldn't. But currently, arch-testers need to make a > > judgement call on what to test when a stable-req bug is filed. Tests run in > > src_test are provided by upstream, and does not guarantee that a package > > that has been merged will actually run on the system. > > If the maintainer could add a couple small scripts to check basic > > functionality > > of the merged package, it would make testing for arch testers much easier > > and reliable. > > Let me give an example. Let's say > > app-misc/screenfetch-2.7.7 is the current stable package for screenfetch in > > the portage tree. > > However, on running, it produces an error on the top, along with the proper > > output. > > If screenfetch-3.0.0 happens to fix that error, maintainer can add a simple > > script > > > > if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then > > eerror "Still producing error" > > fi > > > > To make sure the build is properly updating the screenfetch version, and > > that > > the bug has in fact been fixed. This is the only way I can see to reliabily > > and automatically test packages that have been merged successfully > > > I don't think this needs to be an EAPI change. And if we can acheive > the goal without one, the better. Agreed. > [... lots of interesting stuff ...] Or maintainers and teams could do what the Emacs team does: provide test plans[0]. Naturally, I'd prefer something automated over the very hands-on tests that are a bit of a necessity for an application like Emacs, but they are still better than nothing. Either way, I think it is preferable to have non-upstream tests of whatever shape we pick to be separate from the ebuilds and the source files, for the simple reason that most users won't care about them. Those who do can retrieve them in the same manner as the ATs. And as for my pet peeve, tests that are known to fail, can we also annotate that somehow so I don't waste hours running a test suite that gives zero signal on whether I should add the stable keyword? Even a one-line hin in the stabilization request would be nice. As it is, I keep a list of known-to-fail packages and my testing machinery tells me to not bother with FEATURES=test in those case. Not ideal, but less time-wasty. Regards, Tobias [0] https://wiki.gentoo.org/wiki/Project:Emacs/Test_plans -- if (user_specified) /* Didn't work, but the user is convinced this is the * place. */ linux-2.4.0-test2/drivers/parport/parport_pc.c
