On 04/17/14 15:16, Merlin Moncure wrote:
On Tue, Apr 15, 2014 at 4:47 PM, Josh Berkus <j...@agliodbs.com> wrote:
Hackers,
I think 9.3 has given us evidence that our users aren't giving new
versions of PostgreSQL substantial beta testing, or if they are, they
aren't sharing the results with us.
How can we make beta testing better and more effective? How can we get
more users to actually throw serious workloads at new versions and share
the results?
I've tried a couple of things over the last two years and they haven't
worked all that well. Since we're about to go into another beta testing
period, we need something new. Ideas?
I've seen lots of bugs reported and fixed in the beta period over the
years. My take is that it's basically unrealistic to expect volunteer
beta testers to replace bone fide regression testing.
I think it's a pretty fair statement that we've had some QC issues in
the general area of replication technologies. What this is indicating
to me is that replication needs substantially more coverage in 'make
check'. Since I'm wishing for things, it would be nice to see an
expansion of the buildfarm so that we could [optionally] run various
performance tests as well as various replication scenarios. Then we
could go back to users and say, please donate 'repeatable tests and
machines to run them on' and reap the long term value.
Not at all making light out of any of this...it's a huge project.
The problem with testing replication is that it doesn't fit well into
our standard regression testing. There are way too many moving parts as
well as dependencies on the underlying OS and network topology.
You will discover a ton of race conditions once you actually move from
testing with multiple postmasters (so you can kill one) on the same box
to using multiple virtual machines and instead of completely severing a
network connection using some packet shaping/filtering to introduce
packet loss, limited bandwidth, async routing and so on. At least that
is my experience from throwing that sort of sh*t at Slony at full speed.
Not trying to discourage anyone from trying. Just saying that it doesn't
fit into our existing regression test framework.
Jan
--
Jan Wieck
Senior Software Engineer
http://slony.info
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
