Karl Berry <k...@freefriends.org>, Mon Apr 17 2023 22:16:38 GMT+0200
(Central European Summer Time)
Hi Bogdan,
Then, I analysed the files and added the trick from t/backcompat2.sh
(if possible) and/or removed the extra calls to $ACLOCAL (if possible).
Thanks much for looking into this.
Short version: after a few hours of testing and modifications, I
*may* have saved up to 1 minute and 12 seconds of testing...
Well, at least you get kudos for doing all the research :).
:)
You may look at the attached patch as a result of the investigation
and then ... you're free to completely ignore it :). It works for me,
but I wonder if it won't cause more confusion than it's worth...
I agree. Not worth the complications.
t/backcompat-acout.sh: 35 -> 24s
That seems to me like the only one that might be worth applying the
patch for. Quite a bit more savings than anything else in the list.
Yes. Aclocal is called in a loop here, always with the same set of
(Automake) macros in configure.ac, so it probably always generates the
same aclocal.m4 (no external macros called either). This duplication
can be avoided. Strange that the trick doesn't work in all cases, but
at least it works here.
# A trick to make the test run muuuch faster, by avoiding repeated
# runs of aclocal (one order of magnitude improvement in speed!).
echo 'AC_INIT(x,0) AM_INIT_AUTOMAKE' > configure.ac
Alternatively, I wonder how much this is really saving. Maybe the trick
should not be used anywhere.
5 seconds is the gain, just checked. From about 17.5s to about
12.5s. On the whole test, of course. So, in this particular case, the
saving was meaningful (cut 25-30% of time, even if this is just 5
wallclock seconds).
On the other hand, I'm now calling aclocal 7 times instead of 1, so
6 skipped aclocal calls give about 5s of savings, or about 1s per
aclocal call. In a loop with even tens of iterations the saving would
be visible, but otherwise it's just single seconds, or even literally
1 second, like some of my results show.
- having 1277 .sh files in 't/' means that even if each runs in 30
seconds, you have 10 hours of testing just from the number of tests,
Indeed. The only practical way to run make check is in parallel. I
discovered that early on :). It still takes painfully long for me
(10-15min at best, on a new and fast machine).
I have 4 vcores and I'm afraid the full set would literally take
hours to complete on my machine. 10-15 minutes is a luxury! :)
- it may be better to determine if there are duplicate tests
Sounds awfully hard to do.
I agree. Compare each test with each other. Sometimes within "a
reasonable group" (like tests with the same name and just a number
appended), sometimes with all other tests (like the ones named after a
problem ID). A complexity of n^2 operations :).
My impression is that the (vast?) majority of tests are the direct
result of bug reports. I would not be inclined to tweak, remove, merge,
or change them. Even if two tests are nearly identical, that "nearly"
can make all the difference.
Not sure about the "majority" (I didn't read each and every file),
but I totally agree with the last part. Sometimes some simple change
in one of the files causes some problem to appear and the test
succeeds (or fails if needed) and e.g. not porting the change to a new
"merged" test may cause that we lose the test for an actual problem
without realizing it (in other words, we lose coverage for some part
of the code).
Furthermore, more "atomic" tests allow you to check single
functionalities and give more "atomic" results - you narrow down which
functionality is failing vs. having a file "t/test-everything.sh",
browsing through the log on each failure, and restarting the (hours
long) test after each fix. We don't want that.
That's the "balance slider" here: more tests = more time (no
guarantee that less tests would actually merge something and require
less time), but also more tests = more comfort.
- as you see above, t/pr401b.sh takes 1m42s to run. I wonder if e.g.
running the 'distcheck' target in tests would be the main factor
Sounds very likely to me. Distcheck is inherently a lengthy process. I
can't imagine how it can be sped up. Although I agree that 1:42 seems
rather long for a trivial package like those in the tests.
It runs '$MAKE distcheck' 3 times and one '$MAKE check' + '$MAKE
distclean' pair, so fortunately it's not one 'distcheck' that is
taking so long. Having 25-30s per one 'distcheck' means there isn't a
minute to chop off any longer, but - again - maybe just single
seconds. 25-30s doesn't sound so bad compared to 1:42...
Same case for t/pr401c.sh and t/pr401.sh, although shorter times.
At a glance, I see required='cc libtoolize' in 401b, whereas 401c and
401 only have cc. Testing libtool really is different, and really does
take time. So I'm not sure there's any low-hanging fruit here.
I took a quick look at those 3, and you're probably right. All the
'distcheck' are made on different configurations, so they most
probably must stay as-is.
Thanks again for doing all this work,
:)
--
Regards - Bogdan ('bogdro') D. (GNU/Linux & FreeDOS)
X86 assembly (DOS, GNU/Linux): http://bogdro.evai.pl/index-en.php
Soft(EN): http://bogdro.evai.pl/soft http://bogdro.evai.pl/soft4asm
www.Xiph.org www.TorProject.org www.LibreOffice.org www.GnuPG.org
