On Wed, Sep 14, 2011 at 8:54 AM, Ryan Hadley <r...@sgizmo.com> wrote: > Hi, > > So, here's the backstory: > > We were running Cassandra 0.7.4 and at one point in time had a node in the > ring at 10.84.73.18. We removed this node from the ring successfully in > 0.7.4. It stopped showing in the nodetool ring command. But occasionally we'd > still get weird log entries about failing to write/read to IP 10.84.73.18. > > We upgraded to Cassandra 0.8.4. Now, nodetool ring shows this old node: > > 10.84.73.18 datacenter1 rack1 Down Leaving ? 6.71% > 32695837177645752437561450928649262701 > > So I started a nodetool removetoken on 32695837177645752437561450928649262701 > last Friday. It's still going strong this morning, on day 5: > > ./bin/nodetool -h 10.84.73.47 -p 8080 removetoken status > RemovalStatus: Removing token (32695837177645752437561450928649262701). > Waiting for replication confirmation from > [/10.84.73.49,/10.84.73.48,/10.84.73.51]. > > Should I just be patient? Or is something really weird with this node?
5 days seems excessive unless there is a very large amount of data per node. I would check nodetool netstats, and if the streams don't look active issue a 'removetoken force' against 10.84.73.47 and accept that you may possibly need to run repair to restore the replica count. -Brandon
