On Wed, Mar 23, 2011 at 2:23 PM, Alexis Lê-Quôc <a...@datadoghq.com> wrote:
> I'm going to take a stab at a hypothesis: > Sunday: I drain and decommission 2.3.4.193 *but* I forget to run node > cleanup on the rest of the nodes. The ring looks clean but I did not see > "Annoucing that ..." in the logs. > > Tuesday: the ghost node reappears on the ring for all nodes. > > Could this be caused by old hinted handoffs for 2.3.4.193 that were > processed at that time, causing the rest of the nodes to think that the > 2.3.4.193 is still present (albeit down)? > No. See my hypothesis and patch on https://issues.apache.org/jira/browse/CASSANDRA-2371 The Sun-Tues connection here is revealing, because gossiper's aVeryLongTime is 3 days. Should cleanup be run periodically? I run repair every few days (my > gcgraceperiod is 10 days). > You only need to run cleanup after topology changes. Hints expire themselves at gcgrace. -Brandon
