On 05/06/2016 02:29 PM, Andres Freund wrote:
Hi,
On 2016-05-06 14:17:13 -0700, Joshua D. Drake wrote:
How do I test?
Is there a script I can run?
Unfortunately there's few interesting things to test with pre-made
scripts. There's no relevant OS dependency here, so each already
existing test doesn't really lead to significantly increased coverage
being run by other people. Generally, when testing for correctness
issues, it's often of limited benefit to run tests written by the author
of reviewer - such scripts will usually just test things either has
thought of. The dangerous areas are the ones neither author or reviewer
has considered.
I can't argue with that.
Are there specific things I can do to try and break it?
Upgrade clusters using pg_upgrade and make sure things like index only
scans still work and yield correct data. Set up workloads that involve
freezing, and check that less WAL (and not more!) is generated with 9.6
than with 9.5. Make sure queries still work.
What are we looking for exactly?
Data corruption, efficiency problems.
I am really not trying to be difficult here but Data Corruption is an
easy one... what is the metric we accept as an efficiency problem?
A lot of -hackers seem to forget that although we have 100 -hackers, we have
10000 "consultant/practitioners". Could I read the code and with a weekend
of WTF and -hackers questions figure out what is going on, yes but a lot of
people couldn't and I don't have the time.
I think tests without reading the code are quite sensible and
important. And it perfectly makes sense to ask for information about
what to test. But fundamentally testing is a lot of work, as is writing
and reviewing code; unless you're really really good at destructive
testing, you won't find much in a 15 minute break.
Yes, this is true but with a proper testing framework, I don't need a 15
minute break. I need 1 hour to configure, the rest just "happens" and
reports back.
I have cycles to test, I have team members to help test (as do *lots* of
other people) but sometimes we just get lost in how to help.
You want me (or people like me) to test more? Give us an easy way to
do it.
Useful additional testing and easy just don't go well together. By the
time I have made it easy I've done the testing that's needed.
I don't know that I can agree with this. A proper harness allows you to
execute: go.sh and boom... 2, 4, even 8 hours later you get a report. I
will not argue that it isn't easy to implement but I know it can be done.
Otherwise, we do what we can, which is try and interface on the things that
will directly and immediately affect us (like keywords and syntax).
The amount of bikeshedding on -hackers steals energy and time for
actually working on stuff, including testing. So I have little sympathy
for the amount of bike shedding done.
Insuring a reasonable and thought out interface for users is not bike
shedding, it is at least as important and possibly more important than
any feature we add.
Sincerely,
JD
Greetings,
Andres Freund
--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
