Hello all,
We had upgrade to Cassandra 3.0.0 and later to 3.0.1. That went well.
System seems stable.
However we have a Spring API that uses Cassandra and has a connection pool
using the com.datastax.cassandra driver. We had to upgrade that as well as
it would not work with the latest driver to Cassandra 2.x
Since we had done that we have had an unstable situation with the driver
crashing and thus making the connection pool useless while Tomcat 8.x and
the application running on it does not crash.
I am posting the NIO errors from the logs here.
I wonder if anyone had the same issues or if I should log a bug for it?
It seems that connections crash, do not get cleared and it leads to a
memory leak stack overflow condition.
I am setting my connect() method as follows on my @Repository class:
private void connect() {
if (session == null) {
String contactpoints = env.getProperty("cassandra.contactpoints");
PlainTextAuthProvider plainAuthProvider = new PlainTextAuthProvider(
env.getProperty("cassandra.username"),
env.getProperty("cassandra.password"));
// Probably needs to set Pooling Options etc
// Cluster.builder().getConfiguration().getPoolingOptions()
// Cluster.builder().withReconnectionPolicy(policy)
// Cluster.builder().withRetryPolicy(policy)
cluster = Cluster.builder()
.addContactPoints(contactpoints.split(","))
.withAuthProvider(plainAuthProvider).build();
// connect() without keyspace
// Session cs = cluster.connect();
keyspace = env.getProperty("cassandra.oauthKeyspace");
//session = cluster.connect(keyspace);
// conect without specifying keyspace
session = cluster.connect();
}
}
I will also recheck the manual for the new driver as well as the source
code to see if I can find anything else or if there is a bug. Problem is
that it is hard to replicate sometimes. It will go stable for 2 days and
then crash.
Thanks,
C.
Here is the log errors:
The web application [App1] appears to have started a thread named
[cluster1-connection-reaper-0] but has failed to stop it. This is very
likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
14-Dec-2015 13:31:09.897 WARNING [localhost-startStop-2]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [App1] appears to have started a thread named
[cluster1-scheduled-task-worker-0] but has failed to stop it. This is very
likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
14-Dec-2015 13:31:09.898 WARNING [localhost-startStop-2]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [App1] appears to have started a thread named
[cluster1-nio-worker-0] but has failed to stop it. This is very likely to
create a memory leak. Stack trace of thread:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
java.lang.Thread.run(Thread.java:745)
14-Dec-2015 13:31:09.898 WARNING [localhost-startStop-2]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [App1] appears to have started a thread named
[cluster1-timeouter-0] but has failed to stop it. This is very likely to
create a memory leak. Stack trace of thread:
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:461)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:360)
java.lang.Thread.run(Thread.java:745)
14-Dec-2015 13:31:09.898 WARNING [localhost-startStop-2]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [App1] appears to have started a thread named
[threadDeathWatcher-2-1] but has failed to stop it. This is very likely to
create a memory leak. Stack trace of thread:
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
java.lang.Thread.run(Thread.java:745)
14-Dec-2015 13:31:09.899 WARNING [localhost-startStop-2]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [App1] appears to have started a thread named
[cluster1-nio-worker-1] but has failed to stop it. This is very likely to
create a memory leak. Stack trace of thread:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
java.lang.Thread.run(Thread.java:745)
14-Dec-2015 13:31:09.899 WARNING [localhost-startStop-2]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [App1] appears to have started a thread named
[cluster1-nio-worker-2] but has failed to stop it. This is very likely to
create a memory leak. Stack trace of thread:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
java.lang.Thread.run(Thread.java:745)
14-Dec-2015 13:31:09.899 WARNING [localhost-startStop-2]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [App1] appears to have started a thread named
[cluster1-nio-worker-3] but has failed to stop it. This is very likely to
create a memory leak. Stack trace of thread:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
java.lang.Thread.run(Thread.java:745)
