Thanks for your helps guys,
As you guessed our schema is
{'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3'} AND
durable_writes = false;
Our reads and writes on LOCAL_ONE with each application (now) using its own DC
as its preferred DC
Here is the nodetool status for one of our tables (all tabes are created the
same way)
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN X.0.0.149 14.6 MB 256 100.0%
0f497235-a0bb-4e47-9434-dd0e126aa432 RAC3
UN X.0.0.251 12.33 MB 256 100.0%
a1307717-4b61-4d57-8658-50460d6d54a1 RAC1
UN X.0.0.79 21.54 MB 256 100.0%
f353c8f3-6b7c-483b-ad9a-3d66d469079e RAC2
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN X.0.2.32 18.08 MB 256 100.0%
103a1cb3-6580-44bd-bf97-28ae160e1119 RAC6
UN X.0.2.211 12.46 MB 256 100.0%
8c8dd5ba-806d-43eb-9ee5-af463e443f46 RAC5
UN X.0.2.186 12.58 MB 256 100.0%
aef904ba-aaab-47f1-9bdc-cc1e0c676f61 RAC4
We ran the nodetool repair and cleanup in case the nodes where balanced but
needed cleaning up – this was not the case :(
Steve
From: Alain RODRIGUEZ <[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Wednesday, 13 April 2016 at 14:48
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: Balancing tokens over 2 datacenter
Hi Steve,
As such, all keyspaces and tables where created on DC1.
The effect of this is that all reads are now going to DC1 and ignoring DC2
I think this is not exactly true. When tables are created, they are created on
a specific keyspace, no matter where you send the alter schema command, schema
will propagate to all the datacenters the keyspace is replicated to.
So the question is: Is your keyspace using 'DC1: 3, DC2: 3' as replication
factors? Could you show us the schema and a nodetool status as well?
WE’ve tried doing , nodetool repair / cleanup – but the reads always go to DC1
Trying to do random things is often not a good idea. For example, as each node
holds 100% of the data, cleanup is an expensive no-op :-).
Anyone know how to rebalance the tokens over DC’s?
Yes, I can help on that, but I need to know your current status.
Basically, your(s) keyspace(s) must be using RF of 3 on the 2 DCs as mentioned,
your client to be configured to stick to the DC in their zone (use a DCAware
policy with the DC name + LOCAL_ONE/QUORUM, see Bhuvan's links) and things
should be better.
If you need more detailed help, let us know what is unclear to you and provide
us with 'nodetool status' output and with your schema (at least keyspaces
config).
C*heers,
-----------------------
Alain Rodriguez - [email protected]<mailto:[email protected]>
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2016-04-13 15:32 GMT+02:00 Bhuvan Rawal
<[email protected]<mailto:[email protected]>>:
This could be because of the way you have configured the policy, have a look at
the below links for configuring the policy
https://datastax.github.io/python-driver/api/cassandra/policies.html
http://stackoverflow.com/questions/22813045/ability-to-write-to-a-particular-cassandra-node
Regards,
Bhuvan
On Wed, Apr 13, 2016 at 6:54 PM, Walsh, Stephen
<[email protected]<mailto:[email protected]>> wrote:
Hi there,
So we have 2 datacenter with 3 nodes each.
Replication factor is 3 per DC (so each node has all data)
We have an application in each DC that writes that Cassandra DC.
Now, due to a miss configuration in our application, we saw that our
application in both DC’s where pointing to DC1.
As such, all keyspaces and tables where created on DC1.
The effect of this is that all reads are now going to DC1 and ignoring DC2
WE’ve tried doing , nodetool repair / cleanup – but the reads always go to DC1?
Anyone know how to rebalance the tokens over DC’s?
Regards
Steve
P.S I know about this article
http://www.datastax.com/dev/blog/balancing-your-cassandra-cluster
But its doesn’t answer my question regarding 2 DC’s token balancing
This email (including any attachments) is proprietary to Aspect Software, Inc.
and may contain information that is confidential. If you have received this
message in error, please do not read, copy or forward this message. Please
notify the sender immediately, delete it from your system and destroy any
copies. You may not further disclose or distribute this email or its
attachments.
This email (including any attachments) is proprietary to Aspect Software, Inc.
and may contain information that is confidential. If you have received this
message in error, please do not read, copy or forward this message. Please
notify the sender immediately, delete it from your system and destroy any
copies. You may not further disclose or distribute this email or its
attachments.
