On Fri, Sep 23, 2011 at 11:04 AM, Radim Kolar <h...@sendmail.cz> wrote: > Dne 10.9.2011 21:48, Chris Goffinet napsal(a): >> >> For things like rolling restarts, we do: >> >> disablethrift >> disablegossip >> (...wait for all nodes to see this node go down..) >> drain > > I discovered problem with this advice. > > If i do nodetool drain before killing node nodetool returns just after flush > and stuff disabling is finished on cassandra node. But flush can trigger > possible compaction and if you kill node after drain it will interrupt > compaction in progress resulting in wasted disk space. I am not sure if tmp > files are cleaned on cassandra start.
While it's true that flush/drain may trigger compactions and killing the node just after them will interrupt those compaction, it's not really a problem of flush or drain, in that you always have the risk of interrupting a compaction when killing a node (i.e, even if drain don't happen to trigger a compaction, there may have a compaction that started before the drain and are not yet finished when you kill the node). Besides, if you kill the node just after the drain, the compaction that it has trigger are probably not very advanced yet so are probably the compaction that are the less wasteful to interrupt. Moreover, yes, tmp files are cleaned on restart. So for a rolling restart, which was the case Chris was talking about, this can hardly be called wasted space. Now it is true that it could be a shame to interrupt a compaction that have been running for a long time and is about to finish (so typically not one that has just been triggered by your drain), but you can always check the compaction manager in JMX to see if it's the case before killing the node. -- Sylvain
