On 2024-06-14 Fr 18:11, Jelte Fennema-Nio wrote:
On Fri, 14 Jun 2024 at 17:49, Tom Lane<t...@sss.pgh.pa.us> wrote:
But what I'd really like to see is some comparison of the
language-provided testing facilities that we're proposing
to depend on. Why is pytest (or whatever) better than Test::More?
Some advantages of pytest over Test::More:
1. It's much easier to debug failing tests using the output that
pytest gives. A good example of this is on pytest its homepage[1]
(i.e. it shows the value given to the call to inc in the error)
2. No need to remember a specific comparison DSL
(is/isnt/is_deeply/like/ok/cmp_ok/isa_ok), just put assert in front of
a boolean expression and your output is great (if you want to add a
message too for clarity you can use: assert a == b, "the world is
ending")
3. Very easy to postgres log files on stderr/stdout when a test fails.
This might be possible/easy with Perl too, but we currently don't do
that. So right now for many failures you're forced to traverse the
build/testrun/... directory tree to find the logs.
I see the fact that we stash the output in a file as a feature. Without
it, capturing failure information in the buildfarm client would be more
difficult, especially if there are multiple failures. So this is
actually something I think we would need for any alternative framework.
Maybe we need an environment setting that would output the
regress_log_00whatever file to stderr on failure. That should be pretty
easy to arrange in the END handler for PostgreSQL::Test::Utils.
4. Pytest has autodiscovery of test files and functions, so we
probably wouldn't have to specify all of the exact test files anymore
in the meson.build files.
I find this comment a bit ironic. We don't need to do that with the
Makefiles, and the requirement to do so was promoted as a meson feature
rather than a limitation, ISTR.
Regarding 2, there are ~150 checks that are using a suboptimal way of
testing for a comparison. Mostly a lot that could use "like(..., ...)"
instead of "ok(... ~= ...)"
❯ grep '\bok(.*=' **.pl | wc -l
151
Well, let's fix those. I would be tempted to use cmp_ok() for just about
all of them.
But the fact that Test::More has a handful of test primitives rather
than just one strikes me as a relatively minor complaint.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
