+1, we ran into the same issue. From docs, nodetool drain shuts down ports which then prevent nodetool from working as it needs to talk to cassandra through the ports.
I asked this same question 2 weeks ago but didn't get an answer. I am guessing but I think 1.2.2 seems to fix this a little better so it doesn't complete shut down(ie. I think in 1.2.2 you can still use nodetool after a drain though I need to retest this). A workaround for this issue though is enable your iptables to isolate all nodes and allow now communication in or out on ports 9160 and 7199, do a drain, start cassandra back up while it is isolated and then snapshot the cluster. OR Allow 1.2.2 to read all the commit logs which is what we did (we did snapshot then drain and created hardlinks to the commit log files as well) Take your pick. I have a feeling they are trying to make this better in the newer releases so we can drain and then snapshot though I would have been fine with a snapshot and no need to drain I think as in our testing the drain did NOT drain. It did move everything in the commit log to sstables but the commit log files where still all there and where the same size AND they all got replayed on startup again. I highly suggest to QA all this yourself and us "du –sh /…/…/commitlog" and "du –sh /…/…/data_files" so you get a feel on what you should expect. Dean From: Andrew Bialecki <andrew.biale...@gmail.com<mailto:andrew.biale...@gmail.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Friday, March 8, 2013 8:36 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Nodetool drain automatically shutting down node? Hey all, We're getting ready to upgrade our cluster to 1.2.2 from 1.1.5 and we're testing the upgrade process on our dev cluster. We turned off all client access to the cluster and then ran "nodetool drain" on the first instance with the intention of running "nodetool snapshot" once it finished. However, after running the drain, didn't see any errors, but the Cassandra process was no longer running. Is that expected? From everything I've read it doesn't seem like it, but maybe I'm mistaken. Here's the relevant portion of the log from that node (notice it says it's shutting down the server thread in there): INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,288 StorageService.java (line 774) DRAINING: starting drain process INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,288 CassandraDaemon.java (line 218) Stop listening to thrift clients INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,315 Gossiper.java (line 1133) Announcing shutdown INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:49,318 MessagingService.java (line 534) Waiting for messaging service to quiesce INFO [ACCEPT-ip-10-116-111-143.ec2.internal/10.116.111.143<http://10.116.111.143>] 2013-03-09 03:26:49,319 MessagingService.java (line 690) MessagingService shutting down server thread. INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:49,338 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-Counter1@177255852(14810190/60139556 serialized/live bytes, 243550 ops) INFO [FlushWriter:7] 2013-03-09 03:26:49,338 Memtable.java (line 264) Writing Memtable-Counter1@177255852(14810190/60139556 serialized/live bytes, 243550 ops) INFO [FlushWriter:7] 2013-03-09 03:26:49,899 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-104-Data.db (15204741 bytes) for commitlog position ReplayPosition(segmentId=1362797442799, position=27621115) INFO [CompactionExecutor:11] 2013-03-09 03:26:49,900 CompactionTask.java (line 109) Compacting [SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-102-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-103-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-104-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-101-Data.db')] INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:50,193 StorageService.java (line 774) DRAINED Thanks in advanced for any help. Cheers, Andrew
