Simon Riggs writes: > It's good that the input is fuzzed, but there needs to be a way to re-run > identical fuzzing or a way to backtrack to find what broke. Not much point > finding bugs we can't identify later.
sqlsmith is deterministic and allows re-generating a sequence of random queries with the --seed argument. Finding a testing methodology that ensures a repeatable server-side is a harder problem though. One would have to avoid touching any kind of concurrency, disable autovacuum, autoanalyze and invoke explicit analyzes/vacuums in concert with query generation. Further, one would have to avoid any kind of concurrency while testing. Even then, 1% of the queries run into a statement_timeout due to randomly generated excessive cross joins. If a timeout just barely happens, it might not do so on the repeated run and the deterministic state is gone from then on. I'm afraid this list is not complete yet. I didn't think the effort of creating this kind of clean-room testing was worth it. If reports of failed assertions with backtrace without a recipe to reproduce them are a nuisance, I'll avoid them in the future. regards, Andreas -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
