I have pulled yesterday commits and now I get this "new" error, and the JVM crash disappeared
both in my machine and in Majordodo Jenkins (it should be open to anyone without login) https://dev.majordodo.org/jenkins/job/BookKeeper%20Master/org.apache.bookkeeper$bookkeeper-server/1/testReport/junit/org.apache.bookkeeper.test/BookieClientTest/testWriteGaps/ Sometimes I get this error (which is also on Apache BookKeeper master Jenkins Jobs) Tests run: 33, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 120.151 sec <<< FAILURE! - in org.apache.bookkeeper.replication.AuditorLedgerCheckerTest testDelayedAuditOfLostBookies[1](org.apache.bookkeeper.replication.AuditorLedgerCheckerTest) Time elapsed: 5.146 sec <<< FAILURE! java.lang.AssertionError: audit of lost bookie isn't delayed at org.apache.bookkeeper.replication.AuditorLedgerCheckerTest._testDelayedAuditOfLostBookies(AuditorLedgerCheckerTest.java:345) at org.apache.bookkeeper.replication.AuditorLedgerCheckerTest.testDelayedAuditOfLostBookies(AuditorLedgerCheckerTest.java:367) My laptop is a good Linux 64bit machine Linux xxxxxxx 4.8.6-201.fc24.x86_64 #1 SMP Thu Nov 3 14:38:57 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux with 32 GB or RAM and SSD disks and I'm using Java 8 update 92 java version "1.8.0_92" Java(TM) SE Runtime Environment (build 1.8.0_92-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode) the JVM crash was reproducible even after reboot my machine Enrico 2016-11-17 23:50 GMT+01:00 Sijie Guo <[email protected]>: > I tried to reproduce the issue at my environment. It seems to be okay. Does > anybody else see any problems in current master? > > - Sijie > > On Thu, Nov 17, 2016 at 12:59 PM, Enrico Olivelli <[email protected]> > wrote: > >> Sometimes I get the JVM crash and sometimes the too many open files error. >> Rarely there is no crash and no IOException but the test on the delayed >> auditor fails like on jenkins. >> I never achieve to run the full suite without errors. >> I will run tests on different machines >> >> Enrico >> >> Il gio 17 nov 2016, 21:05 Sijie Guo <[email protected]> ha scritto: >> >> > Interesting, does it constantly fail for you? >> > >> > On Nov 16, 2016 7:29 AM, "Enrico Olivelli" <[email protected]> wrote: >> > >> > > The only way to make tests work again is to configure 4G or RAM for >> > > surefire >> > > >> > > <argLine>-Xmx4G -Djava.net.preferIPv4Stack=true</argLine> >> > > >> > > New memory leaks ? >> > > >> > > some time (with more RAM) I get some "too many open files" error >> > > >> > > testAsyncBookieRecoveryToRandomBookies[4](org.apache. >> bookkeeper.client. >> > > BookieRecoveryTest) >> > > Time elapsed: 0.094 sec <<< ERROR! >> > > org.jboss.netty.channel.ChannelException: Failed to create a selector. >> > > at org.apache.bookkeeper.client.BookieRecoveryTest. >> > > testAsyncBookieRecoveryToRandomBookies(BookieRecoveryTest.java:353) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest. >> > > testAsyncBookieRecoveryToRandomBookies(BookieRecoveryTest.java:353) >> > > >> > > recoverWithoutPasswordInConf[4](org.apache.bookkeeper. >> > > client.BookieRecoveryTest) >> > > Time elapsed: 0.104 sec <<< ERROR! >> > > org.jboss.netty.channel.ChannelException: Failed to create a selector. >> > > at org.apache.bookkeeper.client.BookieRecoveryTest. >> > > recoverWithoutPasswordInConf(BookieRecoveryTest.java:844) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest. >> > > recoverWithoutPasswordInConf(BookieRecoveryTest.java:844) >> > > >> > > testSyncBookieRecoveryToSpecificBookie[4](org.apache. >> bookkeeper.client. >> > > BookieRecoveryTest) >> > > Time elapsed: 0.042 sec <<< ERROR! >> > > org.jboss.netty.channel.ChannelException: Failed to create a selector. >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testSyncBookieRecoveryToRandomBookiesCheckForDupes[4](org. >> > > apache.bookkeeper.client.BookieRecoveryTest) >> > > Time elapsed: 0.027 sec <<< ERROR! >> > > org.jboss.netty.channel.ChannelException: Failed to create a selector. >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testAsyncBookieRecoveryToSpecificBookie[4](org.apache. >> bookkeeper.client. >> > > BookieRecoveryTest) >> > > Time elapsed: 0.018 sec <<< ERROR! >> > > org.jboss.netty.channel.ChannelException: Failed to open a server >> socket. >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.net.SocketException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testBookieRecoveryOnClosedLedgers[4](org.apache.bookkeeper. >> > > client.BookieRecoveryTest) >> > > Time elapsed: 0.015 sec <<< ERROR! >> > > org.jboss.netty.channel.ChannelException: Failed to create a selector. >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > ensurePasswordUsedForOldLedgers[4](org.apache.bookkeeper. >> > > client.BookieRecoveryTest) >> > > Time elapsed: 0.005 sec <<< ERROR! >> > > org.jboss.netty.channel.ChannelException: Failed to create a selector. >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testSyncBookieRecoveryToRandomBookies[5](org.apache.bookkeeper.client. >> > > BookieRecoveryTest) >> > > Time elapsed: 1.108 sec <<< ERROR! >> > > org.jboss.netty.channel.ChannelException: Failed to create a selector. >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testMetadataConflictWithRecovery[5](org.apache.bookkeeper. >> > > client.BookieRecoveryTest) >> > > Time elapsed: 0.101 sec <<< ERROR! >> > > java.io.IOException: Couldn't instantiate >> > > org.apache.zookeeper.ClientCnxnSocketNIO >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testAsyncBookieRecoveryToRandomBookiesNotEnoughBookies[5]( >> > > org.apache.bookkeeper.client.BookieRecoveryTest) >> > > Time elapsed: 0.102 sec <<< ERROR! >> > > java.io.IOException: Couldn't instantiate >> > > org.apache.zookeeper.ClientCnxnSocketNIO >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testBookieRecoveryOnInRecoveryLedger[5](org.apache.bookkeeper.client. >> > > BookieRecoveryTest) >> > > Time elapsed: 0.102 sec <<< ERROR! >> > > java.io.IOException: Couldn't instantiate >> > > org.apache.zookeeper.ClientCnxnSocketNIO >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testBookieRecoveryOnOpenedLedgers[5](org.apache.bookkeeper. >> > > client.BookieRecoveryTest) >> > > Time elapsed: 0.101 sec <<< ERROR! >> > > java.io.IOException: Couldn't instantiate >> > > org.apache.zookeeper.ClientCnxnSocketNIO >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testAsyncBookieRecoveryToRandomBookies[5](org.apache. >> bookkeeper.client. >> > > BookieRecoveryTest) >> > > Time elapsed: 0.102 sec <<< ERROR! >> > > java.io.IOException: Couldn't instantiate >> > > org.apache.zookeeper.ClientCnxnSocketNIO >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > recoverWithoutPasswordInConf[5](org.apache.bookkeeper. >> > > client.BookieRecoveryTest) >> > > Time elapsed: 0.102 sec <<< ERROR! >> > > java.io.IOException: Couldn't instantiate >> > > org.apache.zookeeper.ClientCnxnSocketNIO >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testSyncBookieRecoveryToSpecificBookie[5](org.apache. >> bookkeeper.client. >> > > BookieRecoveryTest) >> > > Time elapsed: 0.101 sec <<< ERROR! >> > > java.io.IOException: Couldn't instantiate >> > > org.apache.zookeeper.ClientCnxnSocketNIO >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testSyncBookieRecoveryToRandomBookiesCheckForDupes[5](org. >> > > apache.bookkeeper.client.BookieRecoveryTest) >> > > Time elapsed: 0.101 sec <<< ERROR! >> > > java.io.IOException: Couldn't instantiate >> > > org.apache.zookeeper.ClientCnxnSocketNIO >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testAsyncBookieRecoveryToSpecificBookie[5](org.apache. >> bookkeeper.client. >> > > BookieRecoveryTest) >> > > Time elapsed: 0.101 sec <<< ERROR! >> > > java.io.IOException: Couldn't instantiate >> > > org.apache.zookeeper.ClientCnxnSocketNIO >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > testBookieRecoveryOnClosedLedgers[5](org.apache.bookkeeper. >> > > client.BookieRecoveryTest) >> > > Time elapsed: 0.102 sec <<< ERROR! >> > > java.io.IOException: Couldn't instantiate >> > > org.apache.zookeeper.ClientCnxnSocketNIO >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > ensurePasswordUsedForOldLedgers[5](org.apache.bookkeeper. >> > > client.BookieRecoveryTest) >> > > Time elapsed: 0.101 sec <<< ERROR! >> > > java.io.IOException: Couldn't instantiate >> > > org.apache.zookeeper.ClientCnxnSocketNIO >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > Caused by: java.io.IOException: Too many open files >> > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( >> > > BookieRecoveryTest.java:110) >> > > >> > > >> > > >> > > -- Enrico >> > > >> > > 2016-11-16 15:27 GMT+01:00 Enrico Olivelli <[email protected]>: >> > > > Hi all, >> > > > after pulling latest commit 2ba9a2c4f7192ec0abaccdfd334025 >> b4ddb9da66 >> > > > Date: Fri Nov 11 18:37:35 2016 -0800 >> > > > BOOKKEEPER-962: Add more journal timing stats >> > > > >> > > > I'm not able to run the full suite of tests (a simple mvn clean >> > install) >> > > > >> > > > The VM crashes >> > > > >> > > > INFO] Total time: 16:50 min >> > > > [INFO] Finished at: 2016-11-16T15:20:46+01:00 >> > > > [INFO] Final Memory: 39M/1137M >> > > > [INFO] ------------------------------------------------------------ >> > > ------------ >> > > > [ERROR] Failed to execute goal >> > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test >> > > > (default-test) on project bookkeeper-server: ExecutionException The >> > > > forked VM terminated without properly saying goodbye. VM crash or >> > > > System.exit called? >> > > > >> > > > I'm trying to find a work-around >> > > > >> > > > >> > > > Enrico >> > > >> > >> -- >> >> >> -- Enrico Olivelli >>
