On 1/17/22 13:19, Andres Freund wrote: > Hi, > > On 2022-01-17 10:25:12 -0500, Andrew Dunstan wrote: >> The buildfarm is moving in the opposite direction, to disaggregate >> steps. > I'm a bit confused as to where you want changes to vcregress.pl > going. Upthread you argued against adding more granular targets to > vcregress. But this seems to be arguing against that?
OK, let me clear that up. Yes I argued upthread for a 'make check-world' equivalent, because it's useful for developers on Windows. But I don't really want to use it in the buildfarm client, where I prefer to run fine-grained tests. > > >> There are several reasons for that, including that it makes for >> less log output that you need to churn through o find out what's gone >> wrong in a particular case, and that it makes disabling certain test >> sets via the buildfarm client's `skip-steps' feature possible. > FWIW, to me this shouldn't require a lot of separate manual test > invocations. And continuing to have lots of granular test invocations from the > buildfarm client is *bad*, because it requires constantly syncing up the set > of test targets. Well, the logic we use for TAP tests is we run them for the following if they have a t subdirectory, subject to configuration settings like PG_TEST_EXTRA: src/test/bin/* contrib/* src/test/modules/* src/test/* As long as any new TAP test gets place in such a location nothing new is required in the buildfarm client. > > It also makes the buildfarm far slower than necessary, because it runs a lot > of stuff serially that it could run in parallel. That's a legitimate concern. I have made some strides towards gross parallelism in the buildfarm by providing for different branches to be run in parallel. This has proven to be fairly successful (i.e. I have not run into any complaints, and several of my own animals utilize it). I can certainly look at doing something of the sort for an individual branch run. > This is particularly a > problem for things like valgrind runs, where individual tests are quite slow - > but just throwing more CPUs at it would help a lot. When I ran a valgrind animal, `make check` was horribly slow, and it's already parallelized. But it was on a VM and I forget how many CPUs the VM had. > > We should set things up so that: > > a) The output of each test can easily be associated with the corresponding set > of log files. > b) The list of tests run can be determined generically by looking at the > filesystems > c) For each test run, it's easy to see whether it failed or succeeded > > As part of the meson stuff (which has its own test runner), I managed to get a > bit towards this by changing the log output hierarchy so that each test gets > its own directory for log files (regress_log_*, initdb.log, postmaster.log, > regression.diffs, server log files). What's missing is a > test.{failed,succeeded} marker or such, to make it easy to report the log > files for just failed tasks. One thing I have been working on is a way to split the log output of an individual buildfarm step, hitherto just a text blob, , so that you can easily navigate to say regress_log_0003-foo-step.log without having to page through myriads of stuff. It's been on the back burner but I need to prioritize it, maybe when the current CF is done. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com