On Sep 14, 2011, at 2:08 PM, Brandon Williams wrote: > On Wed, Sep 14, 2011 at 8:54 AM, Ryan Hadley <r...@sgizmo.com> wrote: >> Hi, >> >> So, here's the backstory: >> >> We were running Cassandra 0.7.4 and at one point in time had a node in the >> ring at 10.84.73.18. We removed this node from the ring successfully in >> 0.7.4. It stopped showing in the nodetool ring command. But occasionally >> we'd still get weird log entries about failing to write/read to IP >> 10.84.73.18. >> >> We upgraded to Cassandra 0.8.4. Now, nodetool ring shows this old node: >> >> 10.84.73.18 datacenter1 rack1 Down Leaving ? 6.71% >> 32695837177645752437561450928649262701 >> >> So I started a nodetool removetoken on >> 32695837177645752437561450928649262701 last Friday. It's still going strong >> this morning, on day 5: >> >> ./bin/nodetool -h 10.84.73.47 -p 8080 removetoken status >> RemovalStatus: Removing token (32695837177645752437561450928649262701). >> Waiting for replication confirmation from >> [/10.84.73.49,/10.84.73.48,/10.84.73.51]. >> >> Should I just be patient? Or is something really weird with this node? > > 5 days seems excessive unless there is a very large amount of data per > node. I would check nodetool netstats, and if the streams don't look > active issue a 'removetoken force' against 10.84.73.47 and accept that > you may possibly need to run repair to restore the replica count. > > -Brandon
Hi Brandon, Thanks for the reply. Quick question though: 1. We write all data to this ring with a TTL of 30 days 2. This node hasn't been in the ring for at least 90 days, more like 120 days since it's been in the ring. So, if I nodetool removetoken forced it, would I still have to be concerned about running a repair? Also, after this node is removed, I'm going to rebalance with nodetool move. Would that remove the repair requirement too? Thanks- Ryan
