On Thu, Apr 05, 2018 at 08:21:15AM +0200, Björn Höfling wrote: > great ideas! > > Last night I did a > > guix pull && guix package -i git > > We have substitutes, right? Yeah, but someone updated git, on my new > machine I didn't configure berlin.guixsd.org yet and hydra didn't have > any substitutes (build wasn't started yet?). > > Building git was relatively fast, but all the tests took ages. And it > was just git. It should work. The git maintainers ran the tests. Marius > when he updated it in commit 5c151862c ran the tests. And that should > be enough of testing. Let's skip the tests.
Not exactly what I am proposing ;). But, even so, I think we should have a switch for turning off tests. Let the builder decide what is good or bad. Too much nannying serves no one. > On the other hand, if I create a new package definition and forget to > run the tests. If upstream is too sloppy, did not run the tests and had > no continuous integration. Who will run the tests then? Hydra should always test before providing a hash that testing is done. > What if I build my package with different sources? > > And you mentioned different environment conditions like machine and > kernel. We still have "only" 70-90% reproducibility. The complement > should have tests enabled. And the question "is my package > reproducible?" is not trivial to answer, and is not computable. Well, I believe that case is overrated and we prove that by actually providing binary substitutes without testing ;) > We saw tests that failed only in 2% of the runs and were fine in 98%. > If we would run those tests "just once", we couldn't figure out that > there is a problem (assuming the problem really is in the software, not > just the tests). > > There could also be practible problems with that: If all write there > software nice and with autoconfigure and we just have a "make && make > test && make install" it's easy to skip the test. But for more > complicated things we have to find a way to tell the build-system how > to skip tests. Totally agree. At this point I patch the tree not to run tests. Pj.
