Hmm, maybe, actually cluster was created not by me.
Another interesting thing was yesterday - by some reason one old node lost one
sstable file(no matter how - thats another problem) and we shut down this node,
clean up all data, and start again. After this result of nodetool status K was
this
That looks like you started the initial nodes with num tokens=1, then later
switched to vnodes, by setting num tokens to 256, then added that new node with
256 vnodes to start. Am I right?
Since you don't have very much data, the easiest way out of this will be to
decommission the original nod
Actually, I have the same doubt. The same happens to me, but I guess it's
because of lack of knowledge in Cassandra vnodes, somehow...
I just added 3 nodes to my old 2 nodes cluster, now I have a 5 nodes
cluster.
As rows should be in a node calculated by HASH / number of nodes, adding a
new node
Hello to everyone!
Please, can someone explain where we made a mistake?
We have cluster with 4 nodes which uses vnodes(256 per node, default settings),
snitch is default on every node: SimpleSnitch.
These four nodes was from beginning of a cluster.
In this cluster we have keyspace with this op
kup plan is to snapshot all data, raise a complete fresh 6 node
> cluster and stream the data using sstable loader. Are there any objections
> about that plan from your point of view?
>
> Thanks in advance!
>
> Andi
>
> From: Aaron
able loader. Are there any objections
about that plan from your point of view?
Thanks in advance!
Andi
From: Aaron Morton [aa...@thelastpickle.com]
Sent: Wednesday, December 18, 2013 3:14 AM
To: Cassandra User
Subject: Re: Unbalanced ring with C* 2.0
> Node: 4 CPU, 6 GB RAM, virtual appliance
>
> Cassandra: 3 GB Heap, vnodes 256
FWIW that’s a very low powered node.
> Maybe we forgot necessary actions during or after cluster expanding process.
> We are open for every idea.
Where the nodes in the seed list when they joined the cluster? If so
Hi,
after adding 2 more nodes to a 4 nodes cluster (before) we are experiencing
high load on both new nodes. After doing some investigation we found out the
following:
- High cpu load on vm5+6
- Higher data load on vm5+6
- Write requests are evenly distributed to all 6 nodes by our client
appl
Check the logs for messages about nodes going up and down, and also look at the
MessagingService MBean for timeouts. If the node in DR 2 times out replying to
DR1 the DR1 node will store a hint.
Also when hints are stored they are TTL'd to the gc_grace_seconds for the CF
(IIRC). If that's low
Here is some more information.
I am running full repair on one of the nodes and I am observing strange
behavior.
Both DCs were up during the data load. But repair is reporting a lot of
out-of-sync data. Why would that be ? Is there a way for me to tell
that WAN may be dropping hinted handoff
Wanted to add one more thing:
I can also tell that the numbers are not consistent across DRs this way
-- I have a column family with really wide rows (a couple million
columns).
DC1 reports higher column counts than DC2. DC2 only becomes consistent
after I do the command a couple of times an
Consider this output from nodetool ring:
Address DC RackStatus State Load
Effective-Ownership Token
127605887595351923798765477786913079396
dc1.5 DC1 RAC1
Actually, doing a nodetool ring is always showing the current node as
owning 99% of the ring
From db-1a-1:
Address DC RackStatus State Load
Effective-Ownership Token
Token(bytes[eaa8])
10.0.4.22 us-east 1a Up
Hello,
We just upgraded from 1.1.2->1.1.9. We utilize the byte ordered
partitioner (we generate our own hashes). We have not yet upgraded
sstables.
Before the upgrade, we had a balanced ring.
After the upgrade, we see:
10.0.4.22 us-east 1a Up Normal 77.66 GB
Maybe people think that 1.2 = Vnodes, when Vnodes are actually not
mandatory and furthermore it is advised to upgrade and then, after a while,
when all is running smooth, eventually switch to vnodes...
2013/2/13 Brandon Williams
> On Tue, Feb 12, 2013 at 6:13 PM, Edward Capriolo
> wrote:
> >
>
On Tue, Feb 12, 2013 at 6:13 PM, Edward Capriolo wrote:
>
> Are vnodes on by default. It seems that many on list are using this feature
> with small clusters.
They are not.
-Brandon
I take that back. vnodes are useful for any size cluster, but I do not see
them as a day one requirement. It seems like many people are stumbling over
this.
On Tuesday, February 12, 2013, Edward Capriolo
wrote:
>
> Are vnodes on by default. It seems that many on list are using this
feature with s
Are vnodes on by default. It seems that many on list are using this feature
with small clusters.
I know these days anything named virtual is sexy, but they are not useful
for small clusters are they. I do not see why people are using them.
On Monday, February 11, 2013, aaron morton wrote:
> So
ase advise the sender immediately by reply
e-mail and delete this message. Thank you for your cooperation.
From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: Monday, February 11, 2013 12:51 PM
To: user@cassandra.apache.org
Subject: Re: unbalanced ring
The tokens are not right, not right at
ware replication,
> your allocation is suspicious.
>
> I’m not sure what you mean by this.
>
> Steve
>
> -Original Message-
> From: Eric Evans [mailto:eev...@acunu.com]
> Sent: Thursday, February 07, 2013 9:56 AM
> To: user@cassandra.apache.org
> Su
> I have about 11M rows of data in this keyspace and none of them are
> exceptionally long … it’s data pulled from Oracle and didn’t include any
> BLOB, etc.
[ ... ]
> From: aaron morton [mailto:aa...@thelastpickle.com]
> Sent: Tuesday, February 05, 2013 3:41 PM
> To: user@ca
.
From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: Tuesday, February 05, 2013 3:41 PM
To: user@cassandra.apache.org
Subject: Re: unbalanced ring
Use nodetool status with vnodes
http://www.datastax.com/dev/blog/upgrading-an-existing-cluster-to-vnodes
The different load can be caused by
Use nodetool status with vnodes
http://www.datastax.com/dev/blog/upgrading-an-existing-cluster-to-vnodes
The different load can be caused by rack affinity, are all the nodes in the
same rack ? Another simple check is have you created some very big rows?
Cheers
-
Aaron Morton
Fre
So I have three nodes in a ring in one data center. My configuration has
num_tokens: 256 set and initial_token commented out. When I look at the ring,
it shows me all of the token ranges of course, and basically identical data for
each range on each node. Here is the Cliff's Notes version of
>> So, if I understand correctly the data of rack1's 5 nodes will be replicated
>> on the single node of rack2.
>> And then, the node of rack1 will host all the data of the cluster.
Yup.
To get RF3 NTS will place a replica in rack 1, then one in rack 2 and then one
in rack 1.
If you are using
On Thu, Dec 20, 2012 at 10:18 AM, DE VITO Dominique
wrote:
> With RF=3 and NetworkTopologyStrategy, "The first replica per data center is
> placed according to the partitioner (same as with SimpleStrategy). Additional
> replicas in the same data center are then determined by walking the ring
>
Hi,
Let's imagine a cluster of 6 nodes, 5 on rack1 and 1 on rack2.
With RF=3 and NetworkTopologyStrategy, "The first replica per data center is
placed according to the partitioner (same as with SimpleStrategy). Additional
replicas in the same data center are then determined by walking the ring
ave received this
message in error, please contact the sender immediately and irrevocably delete
this message and any copies.
From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]
Sent: Thursday, October 11, 2012 09:17
To: user@cassandra.apache.org
Subject: Re: unbalanced ring
Tamar be carefull. Datastax
lain the result above ?
>>>>
>>>> By the way, I have copy the data and import it in a one node dev
>>>> cluster. There I have run a major compaction and the size of my data has
>>>> been significantly reduced (to about 32 GB instead of 133 GB).
>&
ly reduced (to about 32 GB instead of 133 GB).
>>>
>>> How is that possible ?
>>> Do you think that if I run major compaction in both nodes it will
>>> balance the load evenly ?
>>> Should I run major compaction in production ?
>>>
>&g
nd import it in a one node dev cluster.
>> There I have run a major compaction and the size of my data has been
>> significantly reduced (to about 32 GB instead of 133 GB).
>>
>> How is that possible ?
>> Do you think that if I run major compaction in both nodes it will bala
gt; Do you think that if I run major compaction in both nodes it will balance
> the load evenly ?
> Should I run major compaction in production ?
>
> 2012/10/10 Tamar Fraenkel
>
>> Hi!
>> I am re-posting this, now that I have more data and still *unbalanced
>> rin
).
How is that possible ?
Do you think that if I run major compaction in both nodes it will balance
the load evenly ?
Should I run major compaction in production ?
2012/10/10 Tamar Fraenkel
> Hi!
> I am re-posting this, now that I have more data and still *unbalanced ring
> *:
>
>
Hi!
I am re-posting this, now that I have more data and still *unbalanced ring*:
3 nodes,
RF=3, RCL=WCL=QUORUM
Address DC RackStatus State Load
OwnsToken
113427455640312821154458202477256070485
x.x.x.xus-east 1c Up Normal 24.02 GB
It leaves some breathing room for fixing mistakes, adding DCs, etc. The
set of data in a 100 token range is basically the same as a 1 token range:
nothing, statistically speaking.
On Mon, Sep 10, 2012 at 2:21 AM, Guy Incognito wrote:
> out of interest, why -100 and not -1 or + 1? any particul
out of interest, why -100 and not -1 or + 1? any particular reason?
On 06/09/2012 19:17, Tyler Hobbs wrote:
To minimize the impact on the cluster, I would bootstrap a new 1d node
at (42535295865117307932921825928971026432 - 100), then decommission
the 1c node at 4253529586511730793292182592897
To minimize the impact on the cluster, I would bootstrap a new 1d node at
(42535295865117307932921825928971026432 - 100), then decommission the 1c
node at 42535295865117307932921825928971026432 and run cleanup on your
us-east nodes.
On Thu, Sep 6, 2012 at 1:11 PM, William Oberman wrote:
> Didn't
Didn't notice the racks! Of course
If I change a 1c to a 1d, what would I have to do to make sure data
shuffles around correctly? Repair everywhere?
will
On Thu, Sep 6, 2012 at 2:09 PM, Tyler Hobbs wrote:
> The main issue is that one of your us-east nodes is in rack 1d, while the
> resta
The main issue is that one of your us-east nodes is in rack 1d, while the
restart are in rack 1c. With NTS and multiple racks, Cassandra will try
use one node from each rack as a replica for a range until it either meets
the RF for the DC, or runs out of racks, in which case it just picks nodes
se
Hi,
I recently upgraded from 0.8.x to 1.1.x (through 1.0 briefly) and nodetool
-ring seems to have changed from "owns" to "effectively owns".
"Effectively owns" seems to account for replication factor (RF). I'm ok
with all of this, yet I still can't figure out what's up with my cluster.
I have
> Does cleanup only cleanup keys that no longer belong to that node.
Yes.
I guess it could be an artefact of the bulk load. It's not been reported
previously though. Try the cleanup and see how it goes.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelas
Nick, thanks for the response. Does cleanup only cleanup keys that no
longer belong to that node. Just to add more color, when I bulk loaded all
my data into these 6 nodes, all of them had the same amount of data. After
the first nodetool repair, the first node started having more data than the
res
No. Cleanup will scan each sstable to remove data that is no longer
owned by that specific node. It won't compact the sstables together
however.
On Tue, Jun 19, 2012 at 11:11 PM, Raj N wrote:
> But wont that also run a major compaction which is not recommended anymore.
>
> -Raj
>
>
> On Sun, Jun
But wont that also run a major compaction which is not recommended anymore.
-Raj
On Sun, Jun 17, 2012 at 11:58 PM, aaron morton wrote:
> Assuming you have been running repair, it' can't hurt.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thel
Assuming you have been running repair, it' can't hurt.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 17/06/2012, at 4:06 AM, Raj N wrote:
> Nick, do you think I should still run cleanup on the first node.
>
> -Rajesh
>
> On Fri, Jun 15
Nick, do you think I should still run cleanup on the first node.
-Rajesh
On Fri, Jun 15, 2012 at 3:47 PM, Raj N wrote:
> I did run nodetool move. But that was when I was setting up the cluster
> which means I didn't have any data at that time.
>
> -Raj
>
>
> On Fri, Jun 15, 2012 at 1:29 PM, Nic
I did run nodetool move. But that was when I was setting up the cluster
which means I didn't have any data at that time.
-Raj
On Fri, Jun 15, 2012 at 1:29 PM, Nick Bailey wrote:
> Did you start all your nodes at the correct tokens or did you balance
> by moving them? Moving nodes around won't d
Did you start all your nodes at the correct tokens or did you balance
by moving them? Moving nodes around won't delete unneeded data after
the move is done.
Try running 'nodetool cleanup' on all of your nodes.
On Fri, Jun 15, 2012 at 12:24 PM, Raj N wrote:
> Actually I am not worried about the p
Actually I am not worried about the percentage. Its the data I am concerned
about. Look at the first node. It has 102.07GB data. And the other nodes
have around 60 GB(one has 69, but lets ignore that one). I am not
understanding why the first node has almost double the data.
Thanks
-Raj
On Fri, J
This is just a known problem with the nodetool output and multiple
DCs. Your configuration is correct. The problem with nodetool is fixed
in 1.1.1
https://issues.apache.org/jira/browse/CASSANDRA-3412
On Fri, Jun 15, 2012 at 9:59 AM, Raj N wrote:
> Hi experts,
> I have a 6 node cluster across
Hi experts,
I have a 6 node cluster across 2 DCs(DC1:3, DC2:3). I have assigned
tokens using the first strategy(adding 1) mentioned here -
http://wiki.apache.org/cassandra/Operations?#Token_selection
But when I run nodetool ring on my cluster, this is the result I get -
Address DC R
Thanks, I will wait and see as data accumulates.
Thanks,
*Tamar Fraenkel *
Senior Software Engineer, TOK Media
[image: Inline image 1]
ta...@tok-media.com
Tel: +972 2 6409736
Mob: +972 54 8356490
Fax: +972 2 5612956
On Tue, Mar 27, 2012 at 9:00 AM, R. Verlangen wrote:
> Cassandra is
Cassandra is built to store tons and tons of data. In my opinion roughly ~
6MB per node is not enough data to allow it to become a fully balanced
cluster.
2012/3/27 Tamar Fraenkel
> This morning I have
> nodetool ring -h localhost
> Address DC RackStatus State Load
>
This morning I have
nodetool ring -h localhost
Address DC RackStatus State LoadOwns
Token
113427455640312821154458202477256070485
10.34.158.33us-east 1c Up Normal 5.78 MB
33.33% 0
10.38.175.131 us-east 1c Up No
What version are you using?
Anyway try nodetool repair & compact.
maki
2012/3/26 Tamar Fraenkel
> Hi!
> I created Amazon ring using datastax image and started filling the db.
> The cluster seems un-balanced.
>
> nodetool ring returns:
> Address DC RackStatus State Loa
How can I fix this?
add more data. 1.5M is not enough to get reliable reports
Hi!
I created Amazon ring using datastax image and started filling the db.
The cluster seems un-balanced.
nodetool ring returns:
Address DC RackStatus State LoadOwns
Token
113427455640312821154458202477256070485
10.34.158.33us-east 1c
FYI - This manual reordering of the DCs and RACs might make it easier to see
how the tokens are arranged. Pretty sure that the token ranges are picked
correctly. Ignore the Owns column, b/c it is not multi-datacenter aware (so
it thinks all of the nodes are in one ring as opposed to two (DC1 & DC2)
We're running brisk v1 beta2 on 12 nodes - 8 cassandra in DC1 and 4 brisk in DC
2 in EC2. Wrote a few TBs of data to the cluster, and unfortunately the load is
very unbalanced. Every key is the same size and we are using RandomPartitioner.
There are two replicas of data in DC1 and one replica in
59 matches
Mail list logo
