Hi, On 2022-08-17 23:04:20 +0200, Daniel Gustafsson wrote: > Attached is a new version of the pg_regress TAP patch which - per reviewer > feedback - does away with dual output formats and just converts the existing > output to be TAP complaint.
Cool. > while still be TAP compliant enough that running it with prove (with a tiny > Perl wrapper) works. Wonder if we could utilize that for making failures of 002_pg_upgrade.pl and 027_stream_regress.pl easier to understand, by reporting pg_regress' steps as part of the outer test. But that'd probably be at least somewhat hard, due to the embedded test count etc. > This version also strips away the verbose output which these days isn't > terribly useful and mainly consume vertical space. Yay. > Another feature of the patch is to switch error logging to using the common > frontend logging rather than pg_regress rolling its own. The output from > pg_log_error is emitted verbatim by prove as it's on stderr. Sounds good. > I based the support on the original TAP specification and not the new v13 or > v14 since it seemed unclear how well those are supported in testrunners. If > v14 is adopted by testrunners there are some better functionality for > reporting > errors that we could use, but for how this version seems a safer option. Yep, that makes sense. > A normal "make check" with this patch applied now looks like this: > > > +++ regress check in src/test/regress +++ > # running on port 61696 with PID 57910 > ok 1 - test_setup 326 ms > ok 2 - tablespace 401 ms > # parallel group (20 tests): varchar char oid pg_lsn txid int4 regproc money > int2 uuid float4 text name boolean int8 enum float8 bit numeric rangetypes > ok 3 - boolean 129 ms > ok 4 - char 73 ms > ok 5 - name 117 ms > ok 6 - varchar 68 ms > <...snip...> > ok 210 - memoize 137 ms > ok 211 - stats 851 ms > # parallel group (2 tests): event_trigger oidjoins > ok 212 - event_trigger 138 ms > not ok 213 - oidjoins 190 ms > ok 214 - fast_default 149 ms > 1..214 > # 1 of 214 tests failed. > # The differences that caused some tests to fail can be viewed in the > # file "/<path>/src/test/regress/regression.diffs". A copy of the test > summary that you see > # above is saved in the file "/<path>/src/test/regress/regression.out". I'm happy with that compared to our current output. > The ending comment isn't currently shown when executed via prove as it has to > go to STDERR for that to work, and I'm not sure that's something we want to do > in the general case. I don't expect running the pg_regress tests via prove is > going to be the common case. How does the meson TAP ingestion handle > stdout/stderr? It'll parse stdout for tap output and log stderr output separately. > There is a slight reduction in information, for example around tests run > serially vs in a parallel group. This could perhaps be handled by marking the > test name in some way like "tablespace (serial) or prefixing with a symbol or > somerthing. I can take a stab at that in case we think that level of detail > is > important to preserve. I think we could just indent the test "description" for tests in parallel groups: I.e. instead of ok 1 - test_setup 326 ms ok 2 - tablespace 401 ms # parallel group (20 tests): varchar char oid pg_lsn txid int4 regproc money int2 uuid float4 text name boolean int8 enum float8 bit numeric rangetypes ok 3 - boolean 129 ms ok 4 - char 73 ms ok 5 - name 117 ms ok 6 - varchar 68 ms do ok 1 - test_setup 326 ms ok 2 - tablespace 401 ms # parallel group (20 tests): varchar char oid pg_lsn txid int4 regproc money int2 uuid float4 text name boolean int8 enum float8 bit numeric rangetypes ok 3 - boolean 129 ms ok 4 - char 73 ms ok 5 - name 117 ms ok 6 - varchar 68 ms that'd make it sufficiently clear, and is a bit more similar to the current format? I wonder if we should indent the test name so that the number doesn't cause wrapping? And perhaps we can skip the - in that case? I.e. instead of: ok 9 - name 117 ms ok 10 - varchar 68 ms .. ok 99 - something 60 ms ok 100 - memoize 137 ms something like: ok 9 name 117 ms ok 10 varchar 68 ms ... ok 99 something 60 ms ok 100 memoize 137 ms with parallel tests: ok 9 name 117 ms # parallel group (2 tests): varchar varchar2 ok 10 varchar 68 ms ok 11 varchar2 139 ms ok 12 varchar3 44 ms ok 99 something 60 ms ok 100 memoize 137 ms Greetings, Andres Freund
