For a flaky test like this, perhaps running it N times and asserting that it passes ~N-1 iterations?
On 28 February 2018 at 12:00, Otto Fowler <ottobackwa...@gmail.com> wrote: > As long as it is the pr that fails, or that someone with rights to > re-trigger apache travis > notices and restarts the build. > > > On February 28, 2018 at 12:39:03, Gilles (gil...@harfang.homelinux.org) > wrote: > > Hi. > > On Wed, 28 Feb 2018 08:49:42 +0200, Allon Mureinik wrote: > > An alternative approach could be to remove the randomness from the > > test by > > using a predefined random seed and test the overloaded variants that > > accept > > a second java.util.Random argument. This will superficially reduce > > the > > test's coverage, but it does have a reliability advantage, IMHO (as > > seen in > > Otto's original email - these tests do fail occasionally). > > I do not seen the occasional failures as a problem if one is aware > that they can happen due to the "random" nature of the algorithm.[1] > Letting the seed vary (as in the application world) can reveal > unexpected behaviour.[2] > > Gilles > > [1] See here for example: > https://travis-ci.org/apache/commons-rng/builds/345961304 > [2] See here for example: > https://issues.apache.org/jira/projects/MATH/issues/MATH-1361 > > > I posted https://github.com/apache/commons-lang/pull/317 to show this > > approach > > > > On Wed, Feb 28, 2018 at 12:00 AM, Gary Gregory > > <garydgreg...@gmail.com> > > wrote: > > > >> Of course... but how would test then? Shuffle N times and accept a % > >> of > >> non-shuffles? > >> > >> Gary > >> > >> On Tue, Feb 27, 2018, 13:18 Allon Mureinik <murei...@gmail.com> > >> wrote: > >> > >> > There will still be a chance, however infinitesimal, of a failure. > >> :-) > >> > > >> > > >> > On Tue, Feb 27, 2018 at 9:02 PM, Gary Gregory > >> <garydgreg...@gmail.com> > >> > wrote: > >> > > >> > > Why not make the array 1000 items long? > >> > > > >> > > Gary > >> > > > >> > > On Tue, Feb 27, 2018 at 10:31 AM, Allon Mureinik > >> <murei...@gmail.com> > >> > > wrote: > >> > > > >> > > > All the ArrayUtilsTest#testShuffleXYZ tests take an array, > >> shuffle > >> it, > >> > > and > >> > > > assert that the result isn't equal to the original array. > >> > > > This is usually true, but there's a small chance that the > >> shuffled > >> > array > >> > > > will be equal to the original array, and thus the test will > >> fail. > >> This > >> > > > chance is higher for the testShuffleBoolean case where the > >> array > >> > contains > >> > > > ten elements, but only two distinct values (true and false). > >> > > > > >> > > > I've sent a PR to remove these problematic assertions, let's > >> see what > >> > the > >> > > > maintainers think of it: > >> > > > https://github.com/apache/commons-lang/pull/316 > >> > > > > >> > > > > >> > > > On Tue, Feb 27, 2018 at 6:59 PM, Otto Fowler < > >> ottobackwa...@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > Note, this does pass in my personal travis: > >> > > > > > >> https://travis-ci.org/ottobackwards/commons-lang/builds/346806991 > >> > > > > > >> > > > > > >> > > > > On February 27, 2018 at 11:58:24, Otto Fowler ( > >> > ottobackwa...@gmail.com > >> > > ) > >> > > > > wrote: > >> > > > > > >> > > > > My PR is currently failing for java 9 on this test. Anyone > >> have > >> any > >> > > idea > >> > > > > why? > >> > > > > > >> > > > > [INFO] Running org.apache.commons.lang3.ArrayUtilsTest > >> > > > > [ERROR] Tests run: 307, Failures: 1, Errors: 0, Skipped: 0, > >> Time > >> > > elapsed: > >> > > > > 0.114 s <<< FAILURE! - in > >> org.apache.commons.lang3.ArrayUtilsTest > >> > > > > [ERROR] testShuffleBoolean(org.apache. > >> commons.lang3.ArrayUtilsTest) > >> > > Time > >> > > > > elapsed: 0.008 s <<< FAILURE! > >> > > > > java.lang.AssertionError > >> > > > > at > >> > > > > org.apache.commons.lang3.ArrayUtilsTest.testShuffleBoolean( > >> > > > > ArrayUtilsTest.java:5023) > >> > > > > > >> > > > > >> > > > >> > > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > -- Matt Sicker <boa...@gmail.com>
