Are your IPs changing as you restart the cluster? Kubernetes or Mesos or something where your data gets scheduled on different machines? If so, if it gets an IP that was previously in the cluster, it’ll stomp on the old entry in the gossiper maps
-- Jeff Jirsa > On Mar 14, 2019, at 3:42 PM, Fd Habash <fmhab...@gmail.com> wrote: > > I can conclusively say, none of these commands were run. However, I think > this is the likely scenario … > > If you have a cluster of three nodes 1,2,3 … > If 3 shows as DN > Restart C* on 1 & 2 > Nodetool status should NOT show node 3 IP at all. > > Restarting the cluster while a node is down resets gossip state. > > There is a good chance this is what happened. > > Plausible? > > ---------------- > Thank you > > From: Jeff Jirsa > Sent: Thursday, March 14, 2019 11:06 AM > To: cassandra > Subject: Re: Cannot replace_address /10.xx.xx.xx because it doesn't exist > ingossip > > Two things that wouldn't be a bug: > > You could have run removenode > You could have run assassinate > > Also could be some new bug, but that's much less likely. > > > On Thu, Mar 14, 2019 at 2:50 PM Fd Habash <fmhab...@gmail.com> wrote: > I have a node which I know for certain was a cluster member last week. It > showed in nodetool status as DN. When I attempted to replace it today, I got > this message > > ERROR [main] 2019-03-14 14:40:49,208 CassandraDaemon.java:654 - Exception > encountered during startup > java.lang.RuntimeException: Cannot replace_address /10.xx.xx.xxx.xx because > it doesn't exist in gossip > at > org.apache.cassandra.service.StorageService.prepareReplacementInfo(StorageService.java:449) > ~[apache-cassandra-2.2.8.jar:2.2.8] > > > DN 10.xx.xx.xx 388.43 KB 256 6.9% > bdbd632a-bf5d-44d4-b220-f17f258c4701 1e > > Under what conditions does this happen? > > > ---------------- > Thank you > >
