I had logged a bug on JIRA as I think it might be a better way of reporting
this: https://issues.apache.org/jira/browse/CASSANDRA-10861
On Mon, Dec 14, 2015 at 1:52 PM, Carlos A <nando....@gmail.com> wrote:
> Hello all,
>
> We had upgrade to Cassandra 3.0.0 and later to 3.0.1. That went well.
> System seems stable.
>
> However we have a Spring API that uses Cassandra and has a connection pool
> using the com.datastax.cassandra driver. We had to upgrade that as well as
> it would not work with the latest driver to Cassandra 2.x
>
> Since we had done that we have had an unstable situation with the driver
> crashing and thus making the connection pool useless while Tomcat 8.x and
> the application running on it does not crash.
>
> I am posting the NIO errors from the logs here.
>
> I wonder if anyone had the same issues or if I should log a bug for it?
>
> It seems that connections crash, do not get cleared and it leads to a
> memory leak stack overflow condition.
>
> I am setting my connect() method as follows on my @Repository class:
>
>
> private void connect() {
> if (session == null) {
> String contactpoints = env.getProperty("cassandra.contactpoints");
> PlainTextAuthProvider plainAuthProvider = new PlainTextAuthProvider(
> env.getProperty("cassandra.username"),
> env.getProperty("cassandra.password"));
> // Probably needs to set Pooling Options etc
> // Cluster.builder().getConfiguration().getPoolingOptions()
> // Cluster.builder().withReconnectionPolicy(policy)
> // Cluster.builder().withRetryPolicy(policy)
> cluster = Cluster.builder()
> .addContactPoints(contactpoints.split(","))
> .withAuthProvider(plainAuthProvider).build();
> // connect() without keyspace
> // Session cs = cluster.connect();
> keyspace = env.getProperty("cassandra.oauthKeyspace");
> //session = cluster.connect(keyspace);
> // conect without specifying keyspace
> session = cluster.connect();
> }
> }
>
> I will also recheck the manual for the new driver as well as the source
> code to see if I can find anything else or if there is a bug. Problem is
> that it is hard to replicate sometimes. It will go stable for 2 days and
> then crash.
>
> Thanks,
>
> C.
>
> Here is the log errors:
>
>
> The web application [App1] appears to have started a thread named
> [cluster1-connection-reaper-0] but has failed to stop it. This is very
> likely to create a memory leak. Stack trace of thread:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
> 14-Dec-2015 13:31:09.897 WARNING [localhost-startStop-2]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [App1] appears to have started a thread named
> [cluster1-scheduled-task-worker-0] but has failed to stop it. This is very
> likely to create a memory leak. Stack trace of thread:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>
>
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
> 14-Dec-2015 13:31:09.898 WARNING [localhost-startStop-2]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [App1] appears to have started a thread named
> [cluster1-nio-worker-0] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
> java.lang.Thread.run(Thread.java:745)
> 14-Dec-2015 13:31:09.898 WARNING [localhost-startStop-2]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [App1] appears to have started a thread named
> [cluster1-timeouter-0] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
> java.lang.Thread.sleep(Native Method)
>
>
> io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:461)
> io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:360)
> java.lang.Thread.run(Thread.java:745)
> 14-Dec-2015 13:31:09.898 WARNING [localhost-startStop-2]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [App1] appears to have started a thread named
> [threadDeathWatcher-2-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
> java.lang.Thread.sleep(Native Method)
> io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
>
>
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> java.lang.Thread.run(Thread.java:745)
> 14-Dec-2015 13:31:09.899 WARNING [localhost-startStop-2]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [App1] appears to have started a thread named
> [cluster1-nio-worker-1] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
> java.lang.Thread.run(Thread.java:745)
> 14-Dec-2015 13:31:09.899 WARNING [localhost-startStop-2]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [App1] appears to have started a thread named
> [cluster1-nio-worker-2] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
> java.lang.Thread.run(Thread.java:745)
> 14-Dec-2015 13:31:09.899 WARNING [localhost-startStop-2]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [App1] appears to have started a thread named
> [cluster1-nio-worker-3] but has failed to stop it. This is very likely to
> create a memory leak. Stack trace of thread:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
>
>
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
> java.lang.Thread.run(Thread.java:745)
>
>
