While all the token math is helpful, I have to also call out the elephant in the room:
You have not correctly configured Cassandra for production. If you had used the correct endpoint snitch & network topology strategy, you would be able to withstand the complete failure of an entire availability zone at QUORUM, or two if you queried at CL=ONE. You are correct about 256 tokens causing issues, it’s one of the reasons why we recommend 32. I’m curious how things behave going as low as 4, personally, but I haven’t done the math / tested it yet. > On Jan 16, 2018, at 2:02 AM, Kyrylo Lebediev <kyrylo_lebed...@epam.com> wrote: > > ...to me it sounds like 'C* isn't that highly-available by design as it's > declared'. > More nodes in a cluster means higher probability of simultaneous node > failures. > And from high-availability standpoint, looks like situation is made even > worse by recommended setting vnodes=256. > > Need to do some math to get numbers/formulas, but now situation doesn't seem > to be promising. > In case smb from C* developers/architects is reading this message, I'd be > grateful to get some links to calculations of C* reliability based on which > decisions were made. > > Regards, > Kyrill > From: kurt greaves <k...@instaclustr.com> > Sent: Tuesday, January 16, 2018 2:16:34 AM > To: User > Subject: Re: vnodes: high availability > > Yeah it's very unlikely that you will have 2 nodes in the cluster with NO > intersecting token ranges (vnodes) for an RF of 3 (probably even 2). > > If node A goes down all 256 ranges will go down, and considering there are > only 49 other nodes all with 256 vnodes each, it's very likely that every > node will be responsible for some range A was also responsible for. I'm not > sure what the exact math is, but think of it this way: If on each node, any > of its 256 token ranges overlap (it's within the next RF-1 or previous RF-1 > token ranges) on the ring with a token range on node A those token ranges > will be down at QUORUM. > > Because token range assignment just uses rand() under the hood, I'm sure you > could prove that it's always going to be the case that any 2 nodes going down > result in a loss of QUORUM for some token range. > > On 15 January 2018 at 19:59, Kyrylo Lebediev <kyrylo_lebed...@epam.com > <mailto:kyrylo_lebed...@epam.com>> wrote: > Thanks Alexander! > > I'm not a MS in math too) Unfortunately. > > Not sure, but it seems to me that probability of 2/49 in your explanation > doesn't take into account that vnodes endpoints are almost evenly distributed > across all nodes (al least it's what I can see from "nodetool ring" output). > > http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/architecture/architectureDataDistributeDistribute_c.html > > <http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/architecture/architectureDataDistributeDistribute_c.html> > of course this vnodes illustration is a theoretical one, but there no 2 nodes > on that diagram that can be switched off without losing a key range (at > CL=QUORUM). > > That's because vnodes_per_node=8 > Nnodes=6. > As far as I understand, situation is getting worse with increase of > vnodes_per_node/Nnode ratio. > Please, correct me if I'm wrong. > > How would the situation differ from this example by DataStax, if we had a > real-life 6-nodes cluster with 8 vnodes on each node? > > Regards, > Kyrill > > From: Alexander Dejanovski <a...@thelastpickle.com > <mailto:a...@thelastpickle.com>> > Sent: Monday, January 15, 2018 8:14:21 PM > > To: user@cassandra.apache.org <mailto:user@cassandra.apache.org> > Subject: Re: vnodes: high availability > > I was corrected off list that the odds of losing data when 2 nodes are down > isn't dependent on the number of vnodes, but only on the number of nodes. > The more vnodes, the smaller the chunks of data you may lose, and vice versa. > I officially suck at statistics, as expected :) > > Le lun. 15 janv. 2018 à 17:55, Alexander Dejanovski <a...@thelastpickle.com > <mailto:a...@thelastpickle.com>> a écrit : > Hi Kyrylo, > > the situation is a bit more nuanced than shown by the Datastax diagram, which > is fairly theoretical. > If you're using SimpleStrategy, there is no rack awareness. Since vnode > distribution is purely random, and the replica for a vnode will be placed on > the node that owns the next vnode in token order (yeah, that's not easy to > formulate), you end up with statistics only. > > I kinda suck at maths but I'm going to risk making a fool of myself :) > > The odds for one vnode to be replicated on another node are, in your case, > 2/49 (out of 49 remaining nodes, 2 replicas need to be placed). > Given you have 256 vnodes, the odds for at least one vnode of a single node > to exist on another one is 256*(2/49) = 10.4% > Since the relationship is bi-directional (there are the same odds for node B > to have a vnode replicated on node A than the opposite), that doubles the > odds of 2 nodes being both replica for at least one vnode : 20.8%. > > Having a smaller number of vnodes will decrease the odds, just as having more > nodes in the cluster. > (now once again, I hope my maths aren't fully wrong, I'm pretty rusty in that > area...) > > How many queries that will affect is a different question as it depends on > which partition currently exist and are queried in the unavailable token > ranges. > > Then you have rack awareness that comes with NetworkTopologyStrategy : > If the number of replicas (3 in your case) is proportional to the number of > racks, Cassandra will spread replicas in different ones. > In that situation, you can theoretically lose as many nodes as you want in a > single rack, you will still have two other replicas available to satisfy > quorum in the remaining racks. > If you start losing nodes in different racks, we're back to doing maths (but > the odds will get slightly different). > > That makes maintenance predictable because you can shut down as many nodes as > you want in a single rack without losing QUORUM. > > Feel free to correct my numbers if I'm wrong. > > Cheers, > > > > > > On Mon, Jan 15, 2018 at 5:27 PM Kyrylo Lebediev <kyrylo_lebed...@epam.com > <mailto:kyrylo_lebed...@epam.com>> wrote: > Thanks, Rahul. > But in your example, at the same time loss of Node3 and Node6 leads to loss > of ranges N, C, J at consistency level QUORUM. > > As far as I understand in case vnodes > N_nodes_in_cluster and > endpoint_snitch=SimpleSnitch, since: > > 1) "secondary" replicas are placed on two nodes 'next' to the node > responsible for a range (in case of RF=3) > 2) there are a lot of vnodes on each node > 3) ranges are evenly distribusted between vnodes in case of SimpleSnitch, > > we get all physical nodes (servers) having mutually adjacent token rages. > Is it correct? > > At least in case of my real-world ~50-nodes cluster with nvodes=256, RF=3 for > this command: > > nodetool ring | grep '^<ip-prefix>' | awk '{print $1}' | uniq | grep -B2 -A2 > '<ip_of_a_node>' | grep -v '<ip_of_a_node>' | grep -v '^--' | sort | uniq | > wc -l > > returned number which equals to Nnodes -1, what means that I can't switch off > 2 nodes at the same time w/o losing of some keyrange for CL=QUORUM. > > Thanks, > Kyrill > From: Rahul Neelakantan <ra...@rahul.be <mailto:ra...@rahul.be>> > Sent: Monday, January 15, 2018 5:20:20 PM > To: user@cassandra.apache.org <mailto:user@cassandra.apache.org> > Subject: Re: vnodes: high availability > > Not necessarily. It depends on how the token ranges for the vNodes are > assigned to them. For example take a look at this diagram > http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/architecture/architectureDataDistributeDistribute_c.html > > <http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/architecture/architectureDataDistributeDistribute_c.html> > > In the vNode part of the diagram, you will see that Loss of Node 3 and Node > 6, will still not have any effect on Token Range A. But yes if you lose two > nodes that both have Token Range A assigned to them (Say Node 1 and Node 2), > you will have unavailability with your specified configuration. > > You can sort of circumvent this by using the DataStax Java Driver and having > the client recognize a degraded cluster and operate temporarily in downgraded > consistency mode > > http://docs.datastax.com/en/latest-java-driver-api/com/datastax/driver/core/policies/DowngradingConsistencyRetryPolicy.html > > <http://docs.datastax.com/en/latest-java-driver-api/com/datastax/driver/core/policies/DowngradingConsistencyRetryPolicy.html> > > - Rahul > > On Mon, Jan 15, 2018 at 10:04 AM, Kyrylo Lebediev <kyrylo_lebed...@epam.com > <mailto:kyrylo_lebed...@epam.com>> wrote: > Hi, > > Let's say we have a C* cluster with following parameters: > - 50 nodes in the cluster > - RF=3 > - vnodes=256 per node > - CL for some queries = QUORUM > - endpoint_snitch = SimpleSnitch > > Is it correct that 2 any nodes down will cause unavailability of a keyrange > at CL=QUORUM? > > Regards, > Kyrill > > > > -- > ----------------- > Alexander Dejanovski > France > @alexanderdeja > > Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com <http://www.thelastpickle.com/> > -- > ----------------- > Alexander Dejanovski > France > @alexanderdeja > > Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com <http://www.thelastpickle.com/>
