thank you for debugging this! I didn't know about this crazy finalizer at all: I tried to figure it out last night, but i could not see the problem and without a windows machine it was hopeless for me.
On Sun, Sep 14, 2014 at 7:29 AM, Uwe Schindler <[email protected]> wrote: > Hi, > > I know why it fails consequently with the new Java 7 APIs: > FileOutputStream since version 1.0 of Java has a custom finalizer() method, > so the file gets closed on GC, kept until now for backwards compatibility. > But if you the stream with Java 7's NIO.2, the created OutputStream instance > (some non-public internal platform specific impl class) no longer has a > finalizer(), so the file handle is never closed when the OutputStream is > GCed. This is great (f*ck finalizers)! > > So by using NIO.2 we have a very nice "detector" for unclosed file, if people > do broken stuff like "new Properties(new FileInputStream(...))" as seen in > Solr quite often! > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: [email protected] > > >> -----Original Message----- >> From: Uwe Schindler [mailto:[email protected]] >> Sent: Sunday, September 14, 2014 1:16 PM >> To: [email protected] >> Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - >> Build # 4310 - Still Failing! >> >> The problem here is incomplete cleanup: >> >> This test parses the whole algorithm, it then creates an Algorithm instance >> containing all Tasks to execute. The "WriteLineDocsTask" is instantiated, >> which in fact opens the output stream for the line docs file in its ctor. >> But as >> the tasks are never executed, so the Task's close() is never called. >> >> I fixed the whole thing by making Algorithm implement AutoCloseable (not >> Closeable, because it throws not only IOException). I also backported the >> whole stuff to Lucene 4, because this is unrelated to Robert's changes. >> Roberts changes just triggered other code execution paths, so GC did not >> clean up algorithm. >> >> TODO: verify other uses of Algorithm and check that they are correctly >> closed. In fact I have seen this bug sometimes when running tests locally, so >> this was existent since long time, just in most cases GC was faster! >> >> Uwe >> >> ----- >> Uwe Schindler >> H.-H.-Meier-Allee 63, D-28213 Bremen >> http://www.thetaphi.de >> eMail: [email protected] >> >> >> > -----Original Message----- >> > From: Uwe Schindler [mailto:[email protected]] >> > Sent: Sunday, September 14, 2014 12:21 PM >> > To: [email protected] >> > Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - >> > Build # 4310 - Still Failing! >> > >> > This fails reproducible. >> > >> > I will dig! >> > >> > ----- >> > Uwe Schindler >> > H.-H.-Meier-Allee 63, D-28213 Bremen >> > http://www.thetaphi.de >> > eMail: [email protected] >> > >> > > -----Original Message----- >> > > From: Policeman Jenkins Server [mailto:[email protected]] >> > > Sent: Sunday, September 14, 2014 2:48 AM >> > > To: [email protected]; [email protected]; [email protected]; >> > > [email protected]; [email protected]; [email protected] >> > > Subject: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - >> > > Build # >> > > 4310 - Still Failing! >> > > >> > > Build: >> > > http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4310/ >> > > Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops - >> > > XX:+UseConcMarkSweepGC >> > > >> > > 1 tests failed. >> > > FAILED: >> > > >> > >> junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfT >> > > a >> > > sksParse >> > > >> > > Error Message: >> > > Could not remove the following files (in the order of attempts): >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt: >> > > java.nio.file.AccessDeniedException: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt: >> > > java.nio.file.AccessDeniedException: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001: >> > > java.nio.file.DirectoryNotEmptyException: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001 >> > > >> > > Stack Trace: >> > > java.io.IOException: Could not remove the following files (in the >> > > order of >> > > attempts): >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt: >> > > java.nio.file.AccessDeniedException: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt: >> > > java.nio.file.AccessDeniedException: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001: >> > > java.nio.file.DirectoryNotEmptyException: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001 >> > > >> > > at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0) >> > > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288) >> > > at >> > > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(Tes >> > > tR >> > > ul >> > > eTemporaryFilesCleanup.java:126) >> > > at >> > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlwa >> > > ys >> > > (T >> > > estRuleAdapter.java:31) >> > > at >> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S >> > > ta >> > > t >> > > ementAdapter.java:43) >> > > at >> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S >> > > ta >> > > t >> > > ementAdapter.java:36) >> > > at >> > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRul >> > > eA >> > > ss >> > > ertionsRequired.java:43) >> > > at >> > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFa >> > > il >> > > ure >> > > .java:48) >> > > at >> > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Tes >> > > tR >> > > ule >> > > IgnoreAfterMaxFailures.java:65) >> > > at >> > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleI >> > > gn >> > > ore >> > > TestSuites.java:55) >> > > at >> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S >> > > ta >> > > t >> > > ementAdapter.java:36) >> > > at >> > > >> > >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner. >> > > run(ThreadLeakControl.java:365) >> > > at java.lang.Thread.run(Thread.java:745) >> > > >> > > >> > > >> > > >> > > Build Log: >> > > [...truncated 7622 lines...] >> > > [junit4] Suite: >> org.apache.lucene.benchmark.byTask.TestPerfTasksParse >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> Spatial Strategy: >> > > RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:Spa >> > > ti al Context{geo=true, calculator=Haversine, >> > > worldBounds=Rect(minX=- >> > > 180.0,maxX=180.0,minY=- >> > > 90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches) >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 1> ------------> queries: >> > > [junit4] 1> >> > > [junit4] 2> NOTE: test params are: codec=Lucene410: {}, >> > > docValues:{}, >> > > sim=DefaultSimilarity, locale=sq_AL, timezone=America/Yakutat >> > > [junit4] 2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_67 >> (64- >> > > bit)/cpus=2,threads=1,free=120281832,total=140083200 >> > > [junit4] 2> NOTE: All tests run in this JVM: [DocMakerTest, >> > > WriteLineDocTaskTest, AddIndexesTaskTest, StreamUtilsTest, >> > > TrecContentSourceTest, TestQualityRun, PerfTaskTest, >> > > LineDocSourceTest, TestPerfTasksLogic, TestPerfTasksParse] >> > > [junit4] 2> NOTE: reproduce with: ant test - >> > Dtestcase=TestPerfTasksParse >> > > -Dtests.seed=2D34950549928D3F -Dtests.slow=true -Dtests.locale=sq_AL >> > > - Dtests.timezone=America/Yakutat -Dtests.file.encoding=ISO-8859-1 >> > > [junit4] ERROR 0.00s | TestPerfTasksParse (suite) <<< >> > > [junit4] > Throwable #1: java.io.IOException: Could not remove the >> > > following files (in the order of attempts): >> > > [junit4] > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt: >> > > java.nio.file.AccessDeniedException: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt >> > > [junit4] > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt: >> > > java.nio.file.AccessDeniedException: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt >> > > [junit4] > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001: >> > > java.nio.file.DirectoryNotEmptyException: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > >> > >> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask >> > > .TestPerfTasksParse-2D34950549928D3F-001 >> > > [junit4] > at >> > > __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0) >> > > [junit4] > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288) >> > > [junit4] > at java.lang.Thread.run(Thread.java:745) >> > > [junit4] Completed in 3.00s, 3 tests, 1 error <<< FAILURES! >> > > >> > > [...truncated 33 lines...] >> > > BUILD FAILED >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > Windows\build.xml:491: The following error occurred while executing >> > > this >> > > line: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > Windows\build.xml:471: The following error occurred while executing >> > > this >> > > line: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > Windows\build.xml:60: >> > > The following error occurred while executing this line: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra- >> > > targets.xml:39: The following error occurred while executing this line: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > Windows\lucene\build.xml:462: The following error occurred while >> > > executing this line: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > Windows\lucene\common-build.xml:2140: The following error occurred >> > > while executing this line: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > Windows\lucene\module-build.xml:58: The following error occurred >> > > while executing this line: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > Windows\lucene\common-build.xml:1358: The following error occurred >> > > while executing this line: >> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk- >> > > Windows\lucene\common-build.xml:961: There were test failures: 18 >> > > suites, >> > > 93 tests, 1 suite-level error >> > > >> > > Total time: 31 minutes 50 seconds >> > > Build step 'Invoke Ant' marked build as failure [description-setter] >> > > Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops - >> > > XX:+UseConcMarkSweepGC Archiving artifacts Recording test results >> > > Email was triggered for: Failure - Any Sending email for trigger: >> > > Failure - Any >> > > >> > >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] For >> > additional commands, e-mail: [email protected] >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] For additional >> commands, e-mail: [email protected] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
