On 13 January 2015 at 00:53, Phil Steitz <phil.ste...@gmail.com> wrote: > On 1/12/15 5:44 PM, sebb wrote: >> On 12 January 2015 at 22:21, Thomas Neidhart <thomas.neidh...@gmail.com> >> wrote: >>> On 01/12/2015 11:17 PM, Phil Steitz wrote: >>>> On 1/12/15 2:30 PM, Thomas Neidhart wrote: >>>>> On 01/12/2015 10:26 PM, Thomas Neidhart wrote: >>>>>> On 01/12/2015 08:09 PM, Phil Steitz wrote: >>>>>>> On 1/12/15 11:37 AM, sebb wrote: >>>>>>>> On 12 January 2015 at 18:11, Phil Steitz <phil.ste...@gmail.com> wrote: >>>>>>>>> On 1/12/15 10:50 AM, sebb wrote: >>>>>>>>>> On 11 January 2015 at 22:10, Phil Steitz <phil.ste...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>>> On 1/11/15 11:19 AM, Phil Steitz wrote: >>>>>>>>>>>> On 1/10/15 10:49 PM, Phil Steitz wrote: >>>>>>>>>>>>> On 1/9/15 6:09 PM, sebb wrote: >>>>>>>>>>>>>> On 10 January 2015 at 01:01, Phil Steitz <phil.ste...@gmail.com> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> On 1/9/15 5:32 PM, sebb wrote: >>>>>>>>>>>>>>>> On 9 January 2015 at 23:48, sebb <seb...@gmail.com> wrote: >>>>>>>>>>>>>>>>> Of the last 6 runs, only 1 had a problem with unit test >>>>>>>>>>>>>>>>> failures. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> All the builds ran on ubuntu3, apart from the failure which >>>>>>>>>>>>>>>>> ran on H10. >>>>>>>>>>>>>>>>> This may have some bearing on the result; I don't yet know. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I had a quick look at 2 tests that failed: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> SimpleRegressionTest.testPerfect >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> SimpleRegressionTest.testPerfectNegative >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Although the test case has some instance data, these >>>>>>>>>>>>>>>>> particular tests >>>>>>>>>>>>>>>>> do not use any, so it does not look like a concurrency issue >>>>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>>>> unit test itself. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The SimpleRegression class has mutable instance data, but the >>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>> cases create their own instance. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I don't know anything about the math functions involved, but >>>>>>>>>>>>>>>>> it looks >>>>>>>>>>>>>>>>> as though Infinity might result from getSignificance() if >>>>>>>>>>>>>>>>> getSlopeStdErr() returns 0, as the latter is used as a >>>>>>>>>>>>>>>>> divisor. Or if >>>>>>>>>>>>>>>>> the field sumXX is 0 because that is also used as a divisor. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Maybe the H10 host has different floating point hardware? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'll try running some more tests on H10. >>>>>>>>>>>>>>>> the build failed again on H10; exactly the same tests failed >>>>>>>>>>>>>>>> as before: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This test: >>>>>>>>>>>>>>>> https://builds.apache.org/job/Commons%20Math%20H10/1/console >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Previous failure: >>>>>>>>>>>>>>>> https://builds.apache.org/job/Commons%20Math/14/console >>>>>>>>>>>>>>> This is actually a bug. Thanks, sebb (and Jenkins)! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Has been here since 1.x. What is going on is that the data sets >>>>>>>>>>>>>>> used in the test cases are set up to be perfect linear >>>>>>>>>>>>>>> relationships, which should in fact lead to mean square error >>>>>>>>>>>>>>> (and >>>>>>>>>>>>>>> hence slope standard error) equal to 0. The Jenkins box must be >>>>>>>>>>>>>>> getting exact 0. The funny thing is the test is there to >>>>>>>>>>>>>>> validate >>>>>>>>>>>>>>> correct performance for models like this. Its success >>>>>>>>>>>>>>> unfortunately >>>>>>>>>>>>>>> depends on poor precision. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I will open a JIRA for this. I don't think it is a release >>>>>>>>>>>>>>> blocker >>>>>>>>>>>>>>> for 3.4.1, as I am sure you would get the same thing in any >>>>>>>>>>>>>>> earlier >>>>>>>>>>>>>>> version of [math]. >>>>>>>>>>>>>> OK good to know. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'll leave the H10 Jenkins job for now to make it easy to retest. >>>>>>>>>>>>> My first guess here was wrong. The infinities are being handled >>>>>>>>>>>>> correctly for the JDKs I have. Something must be going awry in >>>>>>>>>>>>> the >>>>>>>>>>>>> t distribution cumulative probability computation for +INF on the >>>>>>>>>>>>> box that is failing. Is there a way to find out exactly what JDK >>>>>>>>>>>>> and OS version are being used? >>>>>>>>>>>> I just committed a test that tests the t distribution computations >>>>>>>>>>>> directly. It seems to have run clean; but the other test ran clean >>>>>>>>>>>> too. Is there any way to force the build to use the host that >>>>>>>>>>>> fails? >>>>>>>>>>> I can't make any sense of what is going on with the Jenkins builds. >>>>>>>>>>> Clean runs and then lots of errors. This one explains the >>>>>>>>>>> SimpleRegression "problem" (which is not a problem with that class >>>>>>>>>>> at least) >>>>>>>>>>> >>>>>>>>>>> testCumulativeProbablilityExtremes(org.apache.commons.math3.distribution.TDistributionTest) >>>>>>>>>>> Time elapsed: 0.001 sec <<< FAILURE! >>>>>>>>>>> java.lang.AssertionError: expected:<1.0> but was:<-Infinity> >>>>>>>>>>> at org.junit.Assert.fail(Assert.java:88) >>>>>>>>>>> at org.junit.Assert.failNotEquals(Assert.java:743) >>>>>>>>>>> at org.junit.Assert.assertEquals(Assert.java:494) >>>>>>>>>>> at org.junit.Assert.assertEquals(Assert.java:592) >>>>>>>>>>> at >>>>>>>>>>> org.apache.commons.math3.distribution.TDistributionTest.testCumulativeProbablilityExtremes(TDistributionTest.java:109) >>>>>>>>>>> >>>>>>>>>>> Earlier runs this ran clean. There is nothing non-deterministic >>>>>>>>>>> about this test (or quite a few of the others that randomly seem to >>>>>>>>>>> fail). >>>>>>>>>>> >>>>>>>>>>> I wonder if we have a bad cpu or something somewhere. >>>>>>>>>> AFAICS all the failed builds ran on H10. >>>>>>>>>> >>>>>>>>>> IMO it is consistent; the apparent randomness comes from the fact the >>>>>>>>>> there are several Ubuntu hosts, including H10. >>>>>>>>> Am I reading it / looking at the wrong one, or did this one succeed? >>>>>>>>> >>>>>>>>> https://builds.apache.org/view/All/job/Commons%20Math%20H10/6/ >>>>>>>>> >>>>>>>>> That one was right after I added tests confirming that the t >>>>>>>>> distribution cum prob handles INFs correctly. >>>>>>>> That did run on H10 and did succeed; I'd not noticed that one before. >>>>>>>> >>>>>>>> I think it is still true that the failures have only occurred on H10. >>>>>>>> >>>>>>>> However, the latest one is failing: >>>>>>>> >>>>>>>> https://builds.apache.org/job/Commons%20Math/24/console >>>>>>>> >>>>>>>> This is on H11 - I think that's the first time H11 has been used. >>>>>>>> >>>>>>>> I suppose it's possible that H10 and H11 have a common failing, but it >>>>>>>> seems less likely. >>>>>>>> >>>>>>>> I added a bit more debug - showing the value of sumXX - but that seems >>>>>>>> OK on H11. >>>>>>>> >>>>>>>> I just added a bit more debug. >>>>>>> I am pretty sure the SimpleRegressionTest failure is actually cause >>>>>>> by the same thing causing the t-distribution test to fail (the >>>>>>> reason I added that one). >>>>>>> >>>>>>> One that is more straightforward to chase is this one, which fails >>>>>>> pretty consistently when "bad things happen" >>>>>>> >>>>>>> testExpInf(org.apache.commons.math3.complex.ComplexTest) Time elapsed: >>>>>>> 0.001 sec <<< FAILURE! >>>>>>> java.lang.AssertionError: expected:<0.0> but was:<Infinity> >>>>>>> at org.junit.Assert.fail(Assert.java:88) >>>>>>> at org.junit.Assert.failNotEquals(Assert.java:743) >>>>>>> at org.junit.Assert.assertEquals(Assert.java:494) >>>>>>> at org.junit.Assert.assertEquals(Assert.java:592) >>>>>>> at org.apache.commons.math3.TestUtils.assertSame(TestUtils.java:76) >>>>>>> at org.apache.commons.math3.TestUtils.assertSame(TestUtils.java:84) >>>>>>> at >>>>>>> org.apache.commons.math3.complex.ComplexTest.testExpInf(ComplexTest.java:788) >>>>>>> >>>>>>> I would wager that what is going on here is 0.0 * -INF = INF. >>>>>> The output returned by the debug statements added by sebb is: >>>>>> >>>>>> expReal=Infinity >>>>>> cosImag=0.5403023058681398 >>>>>> sinImag=0.8414709848078965 >>>>>> result=(Infinity, Infinity) >>>>>> >>>>>> while expReal should be -Infinity. >>>>>> >>>>>> of course, Math.exp(Infinity) = Infinity. >>>>> oh stupid mistake, please forget my last post. >>>>> I messed up expReal with the actual real value. >>>> But it should be 0, since expReal should be exp(-INF) >>> just added a few more debug output to the test and the result is: >>> >>> real=-Infinity >>> -real=2147483647 >>> expReal=Infinity >>> >>> according to FastMath.exp(), with these values, the code path should be >>> as follows: >>> >>> if (x < 0.0) { >>> intVal = (int) -x; >>> >>> if (intVal > 746) { >>> if (hiPrec != null) { >>> hiPrec[0] = 0.0; >>> hiPrec[1] = 0.0; >>> } >>> --> return 0.0; >>> } >>> >>> >>> but obviously it doesn't do this. I guess we can only inspect the >>> generated class files for a potential compiler bug. >> That suggests there should be some additional FastMath tests to show >> the underlying error.
Actually there is such a test and it also fails. >> Perhaps compare with the basic Math versions where relevant. > > What Thomas is pointing out is that the code is not executing > correctly, unless we are missing something. This has nothing to do > with FastMath vs. Math. I was just suggesting that it would be worth checking Math to make sure it does not behave the same way. > Did we ever find out what the JDK is? No, but could probably add debug to show the java system variables. > Given the sporadic nature of the failures (different tests failing > different times), Are we sure different tests are failing? I've not checked that. The same hosts fail each time. > I wonder if there is some kind of storage / > filesystem or cpu corruption going on. Do the Jenkins slaves share > a common file system or disk array? Are they virtual hosts? No idea. I just think its suspicious that the failures seem to be repeatable. If a host fails once, it always fails. Hardware failures tend to be more random, and given the test environment it's unlikely that the same memory will be used each time. > Phil >> >>> Thomas >>> >>>> Phil >>>>> Thomas >>>>> >>>>>> Thomas >>>>>> >>>>>>>> I can perhaps change the H10 job to additionally run on H11. >>>>>>>> >>>>>>>> >>>>>>>>> Phi >>>>>>>>>>> Phil >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Phil >>>>>>>>>>>>> Phil >>>>>>>>>>>>>>> Phil >>>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>>>>>>>>> >>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>>>> >>>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>>> For additional commands, e-mail: dev-h...@commons.apache.org >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>> For additional commands, e-mail: dev-h...@commons.apache.org >>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >> For additional commands, e-mail: dev-h...@commons.apache.org >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
