Drain stops listening for connections from client and other nodes, and flushes all the data to disk. The purpose is to get everything into SSTables, so we do not want to process any more writes.
The error is logged at DEBUG as it's not important, just means a thread (the processed gossip) was cancelled. Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 8/03/2013, at 7:58 PM, Andrew Bialecki <andrew.biale...@gmail.com> wrote: > If it's helps, here's the log with debug log statements. Possibly issue with > that exception? > > INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,402 > StorageService.java (line 774) DRAINING: starting drain process > INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,403 > CassandraDaemon.java (line 218) Stop listening to thrift clients > INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,404 > Gossiper.java (line 1133) Announcing shutdown > DEBUG [GossipTasks:1] 2013-03-09 03:54:33,328 > DebuggableThreadPoolExecutor.java (line 190) Task cancelled > java.util.concurrent.CancellationException > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:220) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:182) > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:146) > at > org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.afterExecute(DebuggableScheduledThreadPoolExecutor.java:50) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,406 > StorageService.java (line 776) DRAINING: shutting down MessageService > INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,406 > MessagingService.java (line 534) Waiting for messaging service to quiesce > INFO [ACCEPT-ip-10-116-111-143.ec2.internal/10.116.111.143] 2013-03-09 > 03:54:33,407 MessagingService.java (line 690) MessagingService shutting down > server thread. > DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,408 > StorageService.java (line 776) DRAINING: waiting for streaming > DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,408 > StorageService.java (line 776) DRAINING: clearing mutation stage > DEBUG [Thread-5] 2013-03-09 03:54:33,408 Gossiper.java (line 221) Reseting > version for /10.83.55.44 > DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,409 > StorageService.java (line 776) DRAINING: flushing column families > DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,409 > ColumnFamilyStore.java (line 713) forceFlush requested but everything is > clean in Counter1 > DEBUG [Thread-6] 2013-03-09 03:54:33,410 Gossiper.java (line 221) Reseting > version for /10.80.187.124 > DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,410 > ColumnFamilyStore.java (line 713) forceFlush requested but everything is > clean in Super1 > DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,410 > ColumnFamilyStore.java (line 713) forceFlush requested but everything is > clean in SuperCounter1 > DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,410 > ColumnFamilyStore.java (line 713) forceFlush requested but everything is > clean in Standard1 > INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,510 > StorageService.java (line 774) DRAINED > > On Fri, Mar 8, 2013 at 10:36 PM, Andrew Bialecki <andrew.biale...@gmail.com> > wrote: > Hey all, > > We're getting ready to upgrade our cluster to 1.2.2 from 1.1.5 and we're > testing the upgrade process on our dev cluster. We turned off all client > access to the cluster and then ran "nodetool drain" on the first instance > with the intention of running "nodetool snapshot" once it finished. However, > after running the drain, didn't see any errors, but the Cassandra process was > no longer running. Is that expected? From everything I've read it doesn't > seem like it, but maybe I'm mistaken. > > Here's the relevant portion of the log from that node (notice it says it's > shutting down the server thread in there): > > INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,288 > StorageService.java (line 774) DRAINING: starting drain process > INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,288 > CassandraDaemon.java (line 218) Stop listening to thrift clients > INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,315 > Gossiper.java (line 1133) Announcing shutdown > INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:49,318 > MessagingService.java (line 534) Waiting for messaging service to quiesce > INFO [ACCEPT-ip-10-116-111-143.ec2.internal/10.116.111.143] 2013-03-09 > 03:26:49,319 MessagingService.java (line 690) MessagingService shutting down > server thread. > INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:49,338 > ColumnFamilyStore.java (line 659) Enqueuing flush of > Memtable-Counter1@177255852(14810190/60139556 serialized/live bytes, 243550 > ops) > INFO [FlushWriter:7] 2013-03-09 03:26:49,338 Memtable.java (line 264) > Writing Memtable-Counter1@177255852(14810190/60139556 serialized/live bytes, > 243550 ops) > INFO [FlushWriter:7] 2013-03-09 03:26:49,899 Memtable.java (line 305) > Completed flushing > /var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-104-Data.db > (15204741 bytes) for commitlog position > ReplayPosition(segmentId=1362797442799, position=27621115) > INFO [CompactionExecutor:11] 2013-03-09 03:26:49,900 CompactionTask.java > (line 109) Compacting > [SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-102-Data.db'), > > SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-103-Data.db'), > > SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-104-Data.db'), > > SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-101-Data.db')] > INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:50,193 > StorageService.java (line 774) DRAINED > > Thanks in advanced for any help. > > Cheers, > Andrew >
