Just to add on to your response:
*num_tokens* define the number of vnodes a node can have. Default is 256.
*Initial token* range is predefined (For murmur -2**63 to 2**63-1)
So if you have one node in (does not make sense) cluster with
num_tokens as 256 then you will have 256vnodes. Scaling up will
increase the total number of vnodes linearly with respect to number of
nodes in Cassandra cluster. So for 3 node cluster and num_token as
256, you will have (256*3 vnodes). All vnodes will take some chunk
from the initial taken range.
Regards
Manish
On Tue, Nov 9, 2021 at 5:55 AM Tech Id <tech.login....@gmail.com> wrote:
I will try to give a made-up example to show what I understand.
Let us assume our hash function outputs a number between 1 to 10,000
So hash(primary-key) is between 1 and 10,000
Prior to vnodes, the above 1 to 10k range was split among the nodes.
With vnodes, this 10k range is now split into say 20 overall
vnodes. (simplifying with smaller numbers here).
Hence, vnode is just a logical way to hold 10k/20 = 500 values of
hash(ID)
So vnode1 may hold 1 to 500 values of hash(primary-key), vnode2
may hold 501 to 1000 values of hash(ID) and so on.
Now, each node can hold any vnodes.
So if we have 4 nodes in the above example, each declaring 256
num_tokens, then each of them will get an equal number of vnodes =
20/4 = 5 vnodes each.
The 5 vnodes any node gets, is not contiguous.
So it could be:
Node 1 = vn1, vn3, vn10, vn15, vn20
Node 2 = vn2, vn4, vn5, vn18, vn19
etc
This way, each node now holds small ranges of data, each of those
ranges is called a vnode.
That is what I understand from the docs.
On Mon, Nov 8, 2021 at 4:11 PM Jeff Jirsa <jji...@gmail.com> wrote:
I think your mental model here is trying to map a different db
concept (like elasticsearch shards) to a distributed hash
table that doesnt really map that way.
There's no physical thing as a vnode. Vnode, as a concept, is
"a single node runs multiple tokens and owns multiple ranges".
Multiple ranges are the "vnodes". There's not a block of data
that is a vnode. There's just hosts and ranges they own.
On Mon, Nov 8, 2021 at 4:07 PM Tech Id
<tech.login....@gmail.com> wrote:
Thanks Jeff.
One follow-up question please: Each node specifies num_tokens.
So if there are 4 nodes and each specifies 256 tokens,
then it means together they are responsible for 1024 vnodes.
Now, when a fifth node joins and has num_tokens set to 256
as well, then does the system have 1024+256 = 1280 vnodes?
Or the number of vnodes remains constant in the system and
the nodes just divide that according to their num_token's
weightage?
So in the above example, number of vnodes is say constant
at 1000
With 4 nodes each specifying 256 vnodes, every node in
reality gets 1000/4 = 250 vnodes
With 5 nodes each specifying 256 vnodes, every node gets
1000/5 = 200 vnodes
On Mon, Nov 8, 2021 at 3:33 PM Jeff Jirsa
<jji...@gmail.com> wrote:
When a machine starts for the first time, the joining
node basically chooses a number of tokens (num_tokens)
randomly within the range of the partitioner (for
murmur3, -2**63 to 2**63), and then bootstraps to
claim them.
This is sort of a lie, in newer versions, we try to
make it a bit more deterministic (it tries to ensure
an even distribution), but if you just think of it as
random, it'll make more sense.
The only thing that enforces any meaningful order or
distribution here is a rack-aware snitch, which will
ensure that the RF copies of data land on as many
racks as possible (which is where it may skip some
tokens, if they're found to be on the same rack)
On Mon, Nov 8, 2021 at 3:22 PM Tech Id
<tech.login....@gmail.com> wrote:
Thanks Jeff.
I think what you explained below is before and
after vnodes introduction.
The vnodes part is clear - how each node holds a
small range of tokens and how each node holds a
discontiguous set of vnodes.
1. What is not clear is how each node decided
what vnodes it will get. If it were
contiguous, it would have been easy to
understand (like token range).
2. Also the original question of this thread: If
each node does not replicate all its vnodes to
the same 2 nodes (assume RF=2), then how does
it decide where each of its vnode will be
replicated to?
Maybe the answer to #2 is apparent in #1 answer.
But I would really appreciate if someone can help
me understand the above.
On Mon, Nov 8, 2021 at 2:00 PM Jeff Jirsa
<jji...@gmail.com> wrote:
Vnodes are implemented by giving a single
process multiple tokens.
Tokens ultimately determine which data lives
on which node. When you hash a partition key,
it gives you a token (let's say 570). The 3
processes that own token 57 are the next 3
tokens in the ring ABOVE 570, so if you had
A = 0
B = 1000
C = 2000
D = 3000
E = 4000
The replicas for data for token=570 are B,C,D
When you have vnodes and there's lots of
tokens (from the same small set of 5 hosts),
it'd look closer to:
A = 0
C = 100
A = 300
B = 700
D = 800
B = 1000
D = 1300
C = 1700
B = 1800
C = 2000
E = 2100
B = 2400
A = 2900
D = 3000
E = 4000
In this case, the replicas for token=570 are
B, D and C (it would go B, D, B, D, but we
would de-duplicate the B and D and look for
the next non-B/non-D host.= D at 1700)
If you want to see a view of this in your own
cluster, use `nodetool ring` to see the full
token ring.
There's no desire to enforce a replication
mapping where all data on A is replicated to
the same set of replicas of A, because the
point of vnodes is to give A many distinct
replicas so when you replace A, it can
replicate from "many" other sources (maybe a
dozen, maybe a hundred). This was super
important before 4.0, because each replication
stream was single threaded by SENDER, so
vnodes let you use more than 2-3 cores to
re-replicate (in 4.0, it's still single
threaded, but we avoid a lot of
deserialization so we can saturate a nic with
only a few cores, that was much harder to do
before).
On Mon, Nov 8, 2021 at 1:44 PM Tech Id
<tech.login....@gmail.com> wrote:
Hello,
Going through
https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/architecture/archDataDistributeDistribute.html.
But it is not clear how a node decides
where each of its vnodes will be
replicated to.
As an example from the above page:
1. Why is vnode A present in nodes 1,2 and 5
2. BUT vnode B is present in nodes 1,4 and 6
I realize that the diagram is for
illustration purposes only, but the idea
being conveyed should nevertheless be the
same as I suggested above.
So how come node 1 decides to put A on
itself, 2 and 5 but put B on itself, 4 and 6 ?
Shouldn't there be consistency here such
that all vnodes present on A are
replicated to same set of other nodes?
Any clarifications on that would be
appreciated.
I also understand that different vnodes
are replicated to different nodes for
performance.
But all I want to know is the algorithm
that it uses to put them on different nodes.
Thanks!
