Thanks for all the good info. We have found that running drain first before restarting should always be done, even if there is not much data or I/O.
Also, we've found that node tool drain returns often before it's finished, so it's important to watch the logs (or opscenter) for it and any compaction tasks to complete before restarting. On Apr 25, 2014 1:09 PM, "Jon Haddad" <j...@jonhaddad.com> wrote: > You might want to take a peek at what’s happening in the process via > strace -p or tcpdump. I can’t remember ever waiting an hour for a node to > rejoin. > > > On Apr 25, 2014, at 8:59 AM, Tyler Hobbs <ty...@datastax.com> wrote: > > > On Fri, Apr 25, 2014 at 10:43 AM, Phil Burress > <philburress...@gmail.com>wrote: > >> Thanks. I made a change to a single node and it took almost an hour to >> rejoin the cluster (go from DN to UP in nodetool status). The cluster is >> pretty much idle right now and has a very small dataset. Is that normal? > > > Not unless it had to replay a lot of commitlogs on startup. If you look > at your logs and see that that's the case, you may want to run 'nodetool > drain' before stopping the node. > > > -- > Tyler Hobbs > DataStax <http://datastax.com/> > > >
