On 7/25/05, Michael G Schwern <[EMAIL PROTECTED]> wrote:
> > prove --shuffle --list=5,4,0,1,2,3 t # the shuffle list is predetermined
>
> I'm not sure I see the utility in that last one that significantly beats
> out just reordering the arguments to prove. Do you have a use case? And
> what happens when the number of args is > the list?
The whole point of this option is to allow the reproduction of a
certain order even in a Perl that was not compiled with the same
configuration (mainly with respect to the random number generator). I
patched the POD also with the section below.
************** PART OF THE PATCH
***************************************************
=head1 RECREATABLE SHUFFLED TESTS
Shuffling tests is useful for stressing a test suite with respect
to the independency of execution order. But if a subtle bug is
found with shuffling, there must be a way to reproduce the run.
Enter the C<seed> and C<list> options that go together with
the I<shuffle> option.
The shuffling is controled by the Perl random number seed, that
is reported every time a shuffling is done. For example, if
$ prove -b -D --shuffle 0 1 2 3 4
outputs
# shuffle seed: 87829589
2
3
1
4
0
then
$ prove -b -D -s --seed=87829589 0 1 2 3 4
must repeat the same run every time. This is valid for Perl
interpreters compiled with the same configuration. This is not
strictly necessary, but different configurations of
the pseudo-random generation (different libraries or even different
library versions) and different architectures may give different
results.
In such cases, C<prove> can report the permutation list computed
by shuffling by using the I<debug> option:
$ prove -b -D -d -s seed:79617309 0 1 2 3 4
# $Test::Harness::Switches: -Iblib\arch -Iblib\lib
# shuffle seed: 87829589
# shuffle list: 1,2,0,3,4
1
2
0
3
4
Instead of giving the seed for shuffling, the list can be predetermined
with the C<list> argument.
$ prove -b -D -d -s --list=1,2,0,3,4 0 1 2 3 4
will run the same sequence everywhere, without concern for
differences between random number generators.
************** END ***************************************************
In conclusion, we don't need --list in similar Perl builds, --seed
should be enough. There is also the issue of a very large list of
tests, which was not approached by this patch: it just prints a very
very long line when reporting the shuffle permutation in such cases.
Another issue you mentioned "And what happens when the number of args
is > the list?" points to a verification of the suitability of the
given list as a permutation. To check it out, a sub can be written
which checks that the list is really a permutation of 0..N-1, where N
is the number of tests. Something like this would do
# check($n, @list) returns whether @list is a permutation of 0..$n-1
sub check {
my $n = shift;
my %h;
for (@_) {
return 0 if ($_ < 0 || $_ >= $n);
return 0 if $h{$_}++;
}
return keys %h == $n;
}
I am not sure whether this verification is desirable or practical. In
order to be correct, yes. But trying to reproduce shuffled tests will
always fall apart if the number of test varies or the original order
of the test scripts. There are many involved factors: I count on
perl-qa to help revealing what is worth checking or not, so the patch
can be tuned.
> If I may suggest something. --shuffle should print the seed it used
>
> my $seed = $Seed || int rand(2**$Config{randbits});
> print STDERR "Using seed: $seed\n";
> srand $seed;
>
> that way you can repeat a failed --shuffle test without having to first
> remember to set --seed.
It does this indeed. This is essential for reproducing failed tests.
Instead of writing to STDERR, I included it in the test output like
this
print "# shuffle seed: $shuffle_seed\n";
We cannot afford to have an optional printing of this information or
to lose that information in STDERR. If the test fails, maybe it would
be too late to report the seed that caused the situation.
(Instead of "int rand(2**$Config{randbits})" I used the arbitrary "int
rand(1E8)". Yours is a more clever and portable expression.)
Best regards,
Adriano.
