On Judging the Value of Tests

[Jankirk.Vincent., Jamison]

Dear PG Experts,

I am entirely very new to PG development.
Currently, I’m studying the test suites, and I feel the more experienced PG 
devs here could provide some insights on testing because I could not find 
concrete answers in the Postgres documentations.


1.  How do you judge when a test suite is acceptable to be added to 
Postgres OSS source code? (How do you judge the value of a test suite?)


2.  Besides code coverage test, are there other ways/techniques/tools to 
measure how much value a test would have?

This is coming from a perspective that in the current code coverage 
(https://coverage.postgresql.org/), some functions were covered because the 
function was called/run in some tests, while the function/lines may also not 
necessarily be thoroughly/directly tested so the coverage skips some lines of 
code inside a block.


3.  Is there a standard way of writing tests on the source code that we 
should follow, like when should test be written in TAP/SQL/C formats and how 
long should it be?

①  I know that TAP test is for client program tests, SQL for regression 
tests with sql queries, but there are instances also where tests are also 
written in C like isolation tests, etc. How do we best judge which language is 
preferred to use when writing tests for Postgres components? How long should a 
test be when proposing them to hackers page?


4.  In the src/test/examples directory (which are all libpq tests), why is 
the “examples” directory not included when building postgres? (Why weren't 
these libpq tests added to src/interface/libpq/test or in regression test suite 
instead?) In short, how to know where (in which file/directory in source code) 
to put a test?

I’d like to apologize for the series of questions, but I’d really like to get 
insights from the experienced devs.
It's alright if you don't have answers to all questions as this might take much 
of your time.
Thank you very much.


Regards,
Kirk Jamison

RE: On Judging the Value of Tests

[Jankirk.Vincent., Jamison]

 1.  How do you judge when a test suite is acceptable to be added to
 Postgres OSS source code? (How do you judge the value of a test 
 suite?)
>>>
>>> Make your argument for it, and see if others agree. There's no formal 
>>> process.
>
> Yeah.  Also realize that we've been accreting test cases for ~20 years,
> and there's never been any master plan about what to test.  So there's a 
> lot of individual test cases of varying quality/value, and you shouldn't
> necessarily take any one existing test as gospel. I don't hold that out 
> as a great example of software engineering, but it's the truth.
>>>
>>> Inputs into the decision making process include:
>>> * How much coverage of previously untested functionality it adds
>>> * How much code coverage it adds
>>> * How long the test takes to run, especially considering the slow 
>>> buildfarm boxes and development turnaround time
>>> * Whether the test fits into one of the existing suites we run 
>>> routinely, or requires separate steps
>>> * How much work will be required to maintain the test
>>
>> Even with those factors in mind there is a case-by-case variability.
>> Sometimes it is not worth initializing a new instance with initdb for
>> a small test. Parallelization of tests help on large machines by on 
>> buildfarm members the size of a RPI things get time-consuming.
>> Priority is usually given into things that do not consume much time, 
>> so as any developer can run the full set of tests quickly on a laptop.
>> I have for example a small alias to run the full set of basic tests 
>> (make check-world -j 4 PROVE_FLAGS="-j 4") and this finishes within 
>> two minutes if I recall correctly, TAP tests included. So runtime is 
>> really an important matter.
>
> Also, portability/repeatability.  If it doesn't produce the same answers, 
> with very high probability, across all the platforms we support, it isn't 
> going to last long.

Thank you everyone for your valuable input.
I will take note of all your advice. I think it's because I've noticed that a 
lot of functions and lines are still not covered by tests 
(https://coverage.postgresql.org/), so I feel like I want to work on tests 
first while I'm still learning the PG internals and hacking.

 3.  Is there a standard way of writing tests on the source code that
 we should follow, like when should test be written in TAP/SQL/C 
 formats and how long should it be?
>>>
>>> In general, prefer pg_regress if you can use it. Isolation tests for 
>>> concurrency issues.  TAP tests if you can't write it with pg_regress 
>>> or isolation tester. Test modules only if you really must.
>
> Right.  This ties in closely to the incremental runtime needed for one 
> additional test case.  A new query or two in an existing SQL regression 
> script adds little overhead.  A new TAP test adds quite a bit.

I understand that tests should not take that long when running.
For example, how about when proposing tests that just test
parameter settings of a function? 
Let's say, for example, a libpq main function where in different test cases 
the params are null, empty, invalid, or valid.
Are these test types common or recommended?

 4.  In the src/test/examples directory (which are all libpq tests),
 why is the “examples” directory not included when building postgres?
>>>
>>> Dunno, sorry.
>>
>> Those are mainly present as example programs.
>
> I've always taken those to be more documentation, or sample code, than
> test cases.  I'm not sure why they're under src/test/ at all.

Since these example codes are only documentations, so should these be removed 
in the source code instead? Or should these examples code be revised and be 
moved to a test directory in libpq?

Thanks again!

Regards,
Kirk Jamison