Isn't that obvious http://wiki.apache.org/cassandra/NodeTool check drain
On Mon, Jul 30, 2012 at 11:07 AM, Roshan <codeva...@gmail.com> wrote: > Hi > > As a part of the Cassandra upgrade to 1.1.2 from 1.0.6, I am running > *nodetool drain* node by node to empty the commit logs. When draining a > particular node, that node accepting READ+WRITE request from the clients > and > giving below exceptions. > > 2012-07-30 23:08:18,169 ERROR [Cassandra$Processor] Internal error > processing batch_mutate > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has > shut > down > at > > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) > at > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) > at > > org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:420) > at > > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:308) > at > org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:120) > at > > org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:255) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:194) > at > > org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:638) > at > > org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:589) > at > > org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:597) > at > > org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454) > at > > org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) > at > > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > 2012-07-30 23:08:18,174 ERROR [AbstractCassandraDaemon] Fatal exception in > thread Thread[Thread-6,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has > shut > down > at > > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) > at > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) > at > > org.apache.cassandra.net.MessagingService.receive(MessagingService.java:511) > at > > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:159) > at > > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) > 2012-07-30 23:08:18,177 ERROR [AbstractCassandraDaemon] Fatal exception in > thread Thread[Thread-10,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has > shut > down > at > > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) > at > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) > at > > org.apache.cassandra.net.MessagingService.receive(MessagingService.java:511) > at > > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:159) > at > > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) > 2012-07-30 23:08:18,183 ERROR [Cassandra$Processor] Internal error > processing batch_mutate > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has > shut > down > at > > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) > at > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) > at > > org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:420) > at > > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:308) > at > org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:120) > at > > org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:255) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:194) > at > > org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:638) > at > > org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:589) > at > > org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:597) > at > > org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454) > at > > org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) > at > > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > 2012-07-30 23:08:18,194 ERROR [Cassandra$Processor] Internal error > processing batch_mutate > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has > shut > down > at > > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) > at > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) > at > > org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:420) > at > > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:308) > at > org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:120) > at > > org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:255) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:194) > at > > org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:638) > at > > org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:589) > at > > org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:597) > at > > org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454) > at > > org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) > at > > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > After several second I noticed that the Cassandra process is still not > killed and I killed it manually. > > Why is this strange behavior? How should I need to do the drain without > these exceptions? > > Thanks > > /Roshan > > > > > > -- > View this message in context: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-1-0-6-nodetool-drain-gives-lots-of-batch-mutate-exceptions-tp7581497.html > Sent from the cassandra-u...@incubator.apache.org mailing list archive at > Nabble.com. >
