-Dcassandra.join_ring=false is basically a pre-bootstrap phase that says "this machine is about to join the cluster, but hasn't yet, so don't give it a token"
It's taking advantage of a stable but non-terminal state to let you do things like serve queries without owning data - it's a side effect that works, but it's rough because it wasn't exactly built for this purpose. In this state, you're considered a "fat client" - your presence exists in the ring as a "I'm here, about to join the ring with IP a.b.c.d", and you just conveniently decide not to join the ring. If you go away at any time, the cluster says "cool, no big deal, they didn't join the ring anyway". Your hypothesis is probably mostly right here - it's not so much UP or DOWN, it's "still here" or "gone". Because once the instance is DOWN, it gets removed because it hadn't finished joining. Once it's removed, it can come back and say "Hi, me again, about to join this cluster". But, until it's removed as a fat client, when it comes back and says "Hi, me again, about to join this cluster", cassandra says "not so fast friend, you're already here and we haven't yet given up on you joining". Random aside: There are relatively few people on earth who run like this, so I'm super interested in knowing how it's working for you. Does the PHP client still reconnect on every page load, or does it finally support long lived connections / pooling if you're using something like php-fpm or a fastcgi pool? Are the coordinators/proxies here just to handle a ridiculous number of clients, or is it the cost of connecting that's hurting as you blow up the native thread pool on connect for expensive auth? On Wed, Mar 17, 2021 at 5:44 AM Regis Le Bretonnic < r.lebreton...@meetic-corp.com> wrote: > Hi all, > > > > Following a discussion with our adminsys, I have a very practical question. > > We use cassandra proxies (-Dcassandra.join_ring=false) as coordinators for > PHP clients (a loooooot of PHP clients). > > > > Our problem is that restarting Cassandra on proxies sometimes fails with > the following error : > > > > ERROR [main] 2021-03-16 14:18:46,236 CassandraDaemon.java:803 - Exception > encountered during startup > java.lang.RuntimeException: A node with address > XXXXXXXXXXXXXXXX/10.120.1.XXX already exists, cancelling join. Use > cassandra.replace_address if you want to replace this node. > > > > The node mentioned in the ERROR is the one we are restarting… and the > start fails. Of course doing a manual start after works fine. > > This message doesn’t make sense… hostId didn’t changed for this proxy (I > am sure of me : system.local, IP, hostname, … nothing changed… just the > restart). > > > > What I suppose (we don’t all agree about this) is that, as proxies don’t > have data, they start very quickly. Too quickly for gossip protocol knows > that the node was down. > > Could this ERROR log be explained if the node is still known as UP by > seeds servers if the state of the proxy in gossip protocol is not updated > because stop/start is made too quickly ? > > If this hypothesis seems possible, what reasonable delay (with technical > arguments) should be implemented between stop and start ? > We have ~ 100 proxies and 12 classical Cassandra (4 of them are seeds)… > > Thx in advance > > >
