Sometimes I get the JVM crash and sometimes the too many open files error. Rarely there is no crash and no IOException but the test on the delayed auditor fails like on jenkins. I never achieve to run the full suite without errors. I will run tests on different machines
Enrico Il gio 17 nov 2016, 21:05 Sijie Guo <[email protected]> ha scritto: > Interesting, does it constantly fail for you? > > On Nov 16, 2016 7:29 AM, "Enrico Olivelli" <[email protected]> wrote: > > > The only way to make tests work again is to configure 4G or RAM for > > surefire > > > > <argLine>-Xmx4G -Djava.net.preferIPv4Stack=true</argLine> > > > > New memory leaks ? > > > > some time (with more RAM) I get some "too many open files" error > > > > testAsyncBookieRecoveryToRandomBookies[4](org.apache.bookkeeper.client. > > BookieRecoveryTest) > > Time elapsed: 0.094 sec <<< ERROR! > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > at org.apache.bookkeeper.client.BookieRecoveryTest. > > testAsyncBookieRecoveryToRandomBookies(BookieRecoveryTest.java:353) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest. > > testAsyncBookieRecoveryToRandomBookies(BookieRecoveryTest.java:353) > > > > recoverWithoutPasswordInConf[4](org.apache.bookkeeper. > > client.BookieRecoveryTest) > > Time elapsed: 0.104 sec <<< ERROR! > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > at org.apache.bookkeeper.client.BookieRecoveryTest. > > recoverWithoutPasswordInConf(BookieRecoveryTest.java:844) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest. > > recoverWithoutPasswordInConf(BookieRecoveryTest.java:844) > > > > testSyncBookieRecoveryToSpecificBookie[4](org.apache.bookkeeper.client. > > BookieRecoveryTest) > > Time elapsed: 0.042 sec <<< ERROR! > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testSyncBookieRecoveryToRandomBookiesCheckForDupes[4](org. > > apache.bookkeeper.client.BookieRecoveryTest) > > Time elapsed: 0.027 sec <<< ERROR! > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testAsyncBookieRecoveryToSpecificBookie[4](org.apache.bookkeeper.client. > > BookieRecoveryTest) > > Time elapsed: 0.018 sec <<< ERROR! > > org.jboss.netty.channel.ChannelException: Failed to open a server socket. > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.net.SocketException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testBookieRecoveryOnClosedLedgers[4](org.apache.bookkeeper. > > client.BookieRecoveryTest) > > Time elapsed: 0.015 sec <<< ERROR! > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > ensurePasswordUsedForOldLedgers[4](org.apache.bookkeeper. > > client.BookieRecoveryTest) > > Time elapsed: 0.005 sec <<< ERROR! > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testSyncBookieRecoveryToRandomBookies[5](org.apache.bookkeeper.client. > > BookieRecoveryTest) > > Time elapsed: 1.108 sec <<< ERROR! > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testMetadataConflictWithRecovery[5](org.apache.bookkeeper. > > client.BookieRecoveryTest) > > Time elapsed: 0.101 sec <<< ERROR! > > java.io.IOException: Couldn't instantiate > > org.apache.zookeeper.ClientCnxnSocketNIO > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testAsyncBookieRecoveryToRandomBookiesNotEnoughBookies[5]( > > org.apache.bookkeeper.client.BookieRecoveryTest) > > Time elapsed: 0.102 sec <<< ERROR! > > java.io.IOException: Couldn't instantiate > > org.apache.zookeeper.ClientCnxnSocketNIO > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testBookieRecoveryOnInRecoveryLedger[5](org.apache.bookkeeper.client. > > BookieRecoveryTest) > > Time elapsed: 0.102 sec <<< ERROR! > > java.io.IOException: Couldn't instantiate > > org.apache.zookeeper.ClientCnxnSocketNIO > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testBookieRecoveryOnOpenedLedgers[5](org.apache.bookkeeper. > > client.BookieRecoveryTest) > > Time elapsed: 0.101 sec <<< ERROR! > > java.io.IOException: Couldn't instantiate > > org.apache.zookeeper.ClientCnxnSocketNIO > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testAsyncBookieRecoveryToRandomBookies[5](org.apache.bookkeeper.client. > > BookieRecoveryTest) > > Time elapsed: 0.102 sec <<< ERROR! > > java.io.IOException: Couldn't instantiate > > org.apache.zookeeper.ClientCnxnSocketNIO > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > recoverWithoutPasswordInConf[5](org.apache.bookkeeper. > > client.BookieRecoveryTest) > > Time elapsed: 0.102 sec <<< ERROR! > > java.io.IOException: Couldn't instantiate > > org.apache.zookeeper.ClientCnxnSocketNIO > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testSyncBookieRecoveryToSpecificBookie[5](org.apache.bookkeeper.client. > > BookieRecoveryTest) > > Time elapsed: 0.101 sec <<< ERROR! > > java.io.IOException: Couldn't instantiate > > org.apache.zookeeper.ClientCnxnSocketNIO > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testSyncBookieRecoveryToRandomBookiesCheckForDupes[5](org. > > apache.bookkeeper.client.BookieRecoveryTest) > > Time elapsed: 0.101 sec <<< ERROR! > > java.io.IOException: Couldn't instantiate > > org.apache.zookeeper.ClientCnxnSocketNIO > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testAsyncBookieRecoveryToSpecificBookie[5](org.apache.bookkeeper.client. > > BookieRecoveryTest) > > Time elapsed: 0.101 sec <<< ERROR! > > java.io.IOException: Couldn't instantiate > > org.apache.zookeeper.ClientCnxnSocketNIO > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > testBookieRecoveryOnClosedLedgers[5](org.apache.bookkeeper. > > client.BookieRecoveryTest) > > Time elapsed: 0.102 sec <<< ERROR! > > java.io.IOException: Couldn't instantiate > > org.apache.zookeeper.ClientCnxnSocketNIO > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > ensurePasswordUsedForOldLedgers[5](org.apache.bookkeeper. > > client.BookieRecoveryTest) > > Time elapsed: 0.101 sec <<< ERROR! > > java.io.IOException: Couldn't instantiate > > org.apache.zookeeper.ClientCnxnSocketNIO > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > Caused by: java.io.IOException: Too many open files > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > BookieRecoveryTest.java:110) > > > > > > > > -- Enrico > > > > 2016-11-16 15:27 GMT+01:00 Enrico Olivelli <[email protected]>: > > > Hi all, > > > after pulling latest commit 2ba9a2c4f7192ec0abaccdfd334025b4ddb9da66 > > > Date: Fri Nov 11 18:37:35 2016 -0800 > > > BOOKKEEPER-962: Add more journal timing stats > > > > > > I'm not able to run the full suite of tests (a simple mvn clean > install) > > > > > > The VM crashes > > > > > > INFO] Total time: 16:50 min > > > [INFO] Finished at: 2016-11-16T15:20:46+01:00 > > > [INFO] Final Memory: 39M/1137M > > > [INFO] ------------------------------------------------------------ > > ------------ > > > [ERROR] Failed to execute goal > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test > > > (default-test) on project bookkeeper-server: ExecutionException The > > > forked VM terminated without properly saying goodbye. VM crash or > > > System.exit called? > > > > > > I'm trying to find a work-around > > > > > > > > > Enrico > > > -- -- Enrico Olivelli
