I tried to reproduce the issue at my environment. It seems to be okay. Does anybody else see any problems in current master?
- Sijie On Thu, Nov 17, 2016 at 12:59 PM, Enrico Olivelli <[email protected]> wrote: > Sometimes I get the JVM crash and sometimes the too many open files error. > Rarely there is no crash and no IOException but the test on the delayed > auditor fails like on jenkins. > I never achieve to run the full suite without errors. > I will run tests on different machines > > Enrico > > Il gio 17 nov 2016, 21:05 Sijie Guo <[email protected]> ha scritto: > > > Interesting, does it constantly fail for you? > > > > On Nov 16, 2016 7:29 AM, "Enrico Olivelli" <[email protected]> wrote: > > > > > The only way to make tests work again is to configure 4G or RAM for > > > surefire > > > > > > <argLine>-Xmx4G -Djava.net.preferIPv4Stack=true</argLine> > > > > > > New memory leaks ? > > > > > > some time (with more RAM) I get some "too many open files" error > > > > > > testAsyncBookieRecoveryToRandomBookies[4](org.apache. > bookkeeper.client. > > > BookieRecoveryTest) > > > Time elapsed: 0.094 sec <<< ERROR! > > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > > at org.apache.bookkeeper.client.BookieRecoveryTest. > > > testAsyncBookieRecoveryToRandomBookies(BookieRecoveryTest.java:353) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest. > > > testAsyncBookieRecoveryToRandomBookies(BookieRecoveryTest.java:353) > > > > > > recoverWithoutPasswordInConf[4](org.apache.bookkeeper. > > > client.BookieRecoveryTest) > > > Time elapsed: 0.104 sec <<< ERROR! > > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > > at org.apache.bookkeeper.client.BookieRecoveryTest. > > > recoverWithoutPasswordInConf(BookieRecoveryTest.java:844) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest. > > > recoverWithoutPasswordInConf(BookieRecoveryTest.java:844) > > > > > > testSyncBookieRecoveryToSpecificBookie[4](org.apache. > bookkeeper.client. > > > BookieRecoveryTest) > > > Time elapsed: 0.042 sec <<< ERROR! > > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testSyncBookieRecoveryToRandomBookiesCheckForDupes[4](org. > > > apache.bookkeeper.client.BookieRecoveryTest) > > > Time elapsed: 0.027 sec <<< ERROR! > > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testAsyncBookieRecoveryToSpecificBookie[4](org.apache. > bookkeeper.client. > > > BookieRecoveryTest) > > > Time elapsed: 0.018 sec <<< ERROR! > > > org.jboss.netty.channel.ChannelException: Failed to open a server > socket. > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.net.SocketException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testBookieRecoveryOnClosedLedgers[4](org.apache.bookkeeper. > > > client.BookieRecoveryTest) > > > Time elapsed: 0.015 sec <<< ERROR! > > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > ensurePasswordUsedForOldLedgers[4](org.apache.bookkeeper. > > > client.BookieRecoveryTest) > > > Time elapsed: 0.005 sec <<< ERROR! > > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testSyncBookieRecoveryToRandomBookies[5](org.apache.bookkeeper.client. > > > BookieRecoveryTest) > > > Time elapsed: 1.108 sec <<< ERROR! > > > org.jboss.netty.channel.ChannelException: Failed to create a selector. > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testMetadataConflictWithRecovery[5](org.apache.bookkeeper. > > > client.BookieRecoveryTest) > > > Time elapsed: 0.101 sec <<< ERROR! > > > java.io.IOException: Couldn't instantiate > > > org.apache.zookeeper.ClientCnxnSocketNIO > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testAsyncBookieRecoveryToRandomBookiesNotEnoughBookies[5]( > > > org.apache.bookkeeper.client.BookieRecoveryTest) > > > Time elapsed: 0.102 sec <<< ERROR! > > > java.io.IOException: Couldn't instantiate > > > org.apache.zookeeper.ClientCnxnSocketNIO > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testBookieRecoveryOnInRecoveryLedger[5](org.apache.bookkeeper.client. > > > BookieRecoveryTest) > > > Time elapsed: 0.102 sec <<< ERROR! > > > java.io.IOException: Couldn't instantiate > > > org.apache.zookeeper.ClientCnxnSocketNIO > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testBookieRecoveryOnOpenedLedgers[5](org.apache.bookkeeper. > > > client.BookieRecoveryTest) > > > Time elapsed: 0.101 sec <<< ERROR! > > > java.io.IOException: Couldn't instantiate > > > org.apache.zookeeper.ClientCnxnSocketNIO > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testAsyncBookieRecoveryToRandomBookies[5](org.apache. > bookkeeper.client. > > > BookieRecoveryTest) > > > Time elapsed: 0.102 sec <<< ERROR! > > > java.io.IOException: Couldn't instantiate > > > org.apache.zookeeper.ClientCnxnSocketNIO > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > recoverWithoutPasswordInConf[5](org.apache.bookkeeper. > > > client.BookieRecoveryTest) > > > Time elapsed: 0.102 sec <<< ERROR! > > > java.io.IOException: Couldn't instantiate > > > org.apache.zookeeper.ClientCnxnSocketNIO > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testSyncBookieRecoveryToSpecificBookie[5](org.apache. > bookkeeper.client. > > > BookieRecoveryTest) > > > Time elapsed: 0.101 sec <<< ERROR! > > > java.io.IOException: Couldn't instantiate > > > org.apache.zookeeper.ClientCnxnSocketNIO > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testSyncBookieRecoveryToRandomBookiesCheckForDupes[5](org. > > > apache.bookkeeper.client.BookieRecoveryTest) > > > Time elapsed: 0.101 sec <<< ERROR! > > > java.io.IOException: Couldn't instantiate > > > org.apache.zookeeper.ClientCnxnSocketNIO > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testAsyncBookieRecoveryToSpecificBookie[5](org.apache. > bookkeeper.client. > > > BookieRecoveryTest) > > > Time elapsed: 0.101 sec <<< ERROR! > > > java.io.IOException: Couldn't instantiate > > > org.apache.zookeeper.ClientCnxnSocketNIO > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > testBookieRecoveryOnClosedLedgers[5](org.apache.bookkeeper. > > > client.BookieRecoveryTest) > > > Time elapsed: 0.102 sec <<< ERROR! > > > java.io.IOException: Couldn't instantiate > > > org.apache.zookeeper.ClientCnxnSocketNIO > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > ensurePasswordUsedForOldLedgers[5](org.apache.bookkeeper. > > > client.BookieRecoveryTest) > > > Time elapsed: 0.101 sec <<< ERROR! > > > java.io.IOException: Couldn't instantiate > > > org.apache.zookeeper.ClientCnxnSocketNIO > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > Caused by: java.io.IOException: Too many open files > > > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp( > > > BookieRecoveryTest.java:110) > > > > > > > > > > > > -- Enrico > > > > > > 2016-11-16 15:27 GMT+01:00 Enrico Olivelli <[email protected]>: > > > > Hi all, > > > > after pulling latest commit 2ba9a2c4f7192ec0abaccdfd334025 > b4ddb9da66 > > > > Date: Fri Nov 11 18:37:35 2016 -0800 > > > > BOOKKEEPER-962: Add more journal timing stats > > > > > > > > I'm not able to run the full suite of tests (a simple mvn clean > > install) > > > > > > > > The VM crashes > > > > > > > > INFO] Total time: 16:50 min > > > > [INFO] Finished at: 2016-11-16T15:20:46+01:00 > > > > [INFO] Final Memory: 39M/1137M > > > > [INFO] ------------------------------------------------------------ > > > ------------ > > > > [ERROR] Failed to execute goal > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test > > > > (default-test) on project bookkeeper-server: ExecutionException The > > > > forked VM terminated without properly saying goodbye. VM crash or > > > > System.exit called? > > > > > > > > I'm trying to find a work-around > > > > > > > > > > > > Enrico > > > > > > -- > > > -- Enrico Olivelli >
