> On Jan 7, 2019, at 8:23 AM, Jeff Jirsa <jji...@gmail.com> wrote: > > > > >> On Jan 7, 2019, at 6:37 AM, Jonathan Ballet <jbal...@edgelab.ch> wrote: >> >> Hi, >> >> I'm trying to understand how seed nodes are working, when and how do they >> play a part in a Cassandra cluster, and how they should be managed and >> propagated to other nodes. >> >> I have a cluster of 6 Cassandra nodes (let's call them #1 to #6), on which >> node #1 and #2 are seeds. All the configuration files of all the Cassandra >> nodes are currently configured with: >> >> ``` >> seed_provider: >> - class_name: org.apache.cassandra.locator.SimpleSeedProvider >> parameters: >> - seeds: 'IP #1,IP #2' >> ``` >> >> We are using a service discovery tool (Consul) which automatically registers >> new Cassandra nodes with its dedicated health-check and are able to generate >> new configuration based on the content of the service discovery status (with >> Consul-Template). >> >> >> I'm working on how we could improve the upgrades of our servers and how to >> replace them completely (new instance with a new IP address). >> What I would like to do is to replace the machines holding our current seeds >> (#1 and #2 at the moment) in a rolling upgrade fashion, on a regular basis: >> >> * Is it possible to "promote" any non-seed node as a seed node? >> > > Yes - generally you can make any node a seed if you want > >> * Is it possible to "promote" a new seed node without having to restart all >> the nodes? > > nodetool reloadseeds
This is apparently in 4.0+ https://issues.apache.org/jira/browse/CASSANDRA-14190 > > There are a few weird edge cases where seeds are reloaded automatically and > we don’t document how or why (it’s a side effect of an error condition in > hosts going up/down, but it’s generally pretty minor unless your seed > provider is broken) > > > (Also true that you could write a seed provider that did this automatically) > > >> In essence, in my example that would be: >> >> - decide that #2 and #3 will be the new seed nodes >> - update all the configuration files of all the nodes to write the IP >> addresses of #2 and #3 >> - DON'T restart any node - the new seed configuration will be picked up >> only if the Cassandra process restarts >> >> * If I can manage to sort my Cassandra nodes by their age, could it be a >> strategy to have the seeds set to the 2 oldest nodes in the cluster? (This >> implies these nodes would change as the cluster's nodes get >> upgraded/replaced). > > You could do this, seems like a lot of headache for little benefit. Could be > done with simple seed provider and config management (puppet/chef/ansible) > laying down new yaml or with your own seed provider > >> >> >> I also have some more general questions about seed nodes and how they work: >> >> * I understand that seed nodes are used when a node starts and needs to >> discover the rest of the cluster's nodes. Once the node has joined and the >> cluster is stable, are seed nodes still playing a role in day to day >> operations? > > They’re used probabilistically in gossip to encourage convergence. Mostly > useful in large clusters. > >> >> * The documentation says multiple times that not all nodes should be seed >> nodes, but I didn't really find any place about the consequences it has to >> have "too many" seed nodes. > > Decreases effectiveness of probabilistic gossiping with seed for convergence > >> Also, relatively to the questions I asked above, is there any downsides of >> having changing seed nodes in a cluster? (with the exact same, at some point >> I define #1 and #2 to be seeds, then later #4 and #5, etc.) >> > > No > >> >> Thanks for helping me to understand better how seeds are working! >> >> Jonathan >>
