Hi Sean,
Are you using any rack aware topology? --> we are using gossip file
Are you using any rack aware topology? --> we are using gossip file What are
your partition keys? --> Partition key is uniqIs it possible that your
partition keys do not divide up as cleanly as you would like across the cluster
because the data is not evenly distributed (by partition key)? --> No, we
verified it.
If it was partition key issue, we would see similar number of partition keys
across nodes. If we look closely number of keys across nodes vary a lot.
Number of partitions (estimate): 3142552Number of partitions (estimate):
15625442Number of partitions (estimate): 15244021Number of partitions
(estimate): 9592992Number of partitions (estimate): 15839280
On Monday, June 18, 2018, 5:39:08 PM EDT, Durity, Sean R
<[email protected]> wrote:
#yiv2571733579 #yiv2571733579 -- _filtered #yiv2571733579
{font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv2571733579
{panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv2571733579
{font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}#yiv2571733579
#yiv2571733579 p.yiv2571733579MsoNormal, #yiv2571733579
li.yiv2571733579MsoNormal, #yiv2571733579 div.yiv2571733579MsoNormal
{margin:0in;margin-bottom:.0001pt;font-size:12.0pt;font-family:New
serif;}#yiv2571733579 a:link, #yiv2571733579 span.yiv2571733579MsoHyperlink
{color:#0563C1;text-decoration:underline;}#yiv2571733579 a:visited,
#yiv2571733579 span.yiv2571733579MsoHyperlinkFollowed
{color:#954F72;text-decoration:underline;}#yiv2571733579 p
{margin-right:0in;margin-left:0in;font-size:12.0pt;font-family:New
serif;}#yiv2571733579 p.yiv2571733579msonormal0, #yiv2571733579
li.yiv2571733579msonormal0, #yiv2571733579 div.yiv2571733579msonormal0
{margin-right:0in;margin-left:0in;font-size:12.0pt;font-family:New
serif;}#yiv2571733579 p.yiv2571733579ydpe3237830p1, #yiv2571733579
li.yiv2571733579ydpe3237830p1, #yiv2571733579 div.yiv2571733579ydpe3237830p1
{margin-right:0in;margin-left:0in;font-size:12.0pt;font-family:New
serif;}#yiv2571733579 span.yiv2571733579ydpe3237830s1 {}#yiv2571733579
span.yiv2571733579ydpe3237830apple-converted-space {}#yiv2571733579
span.yiv2571733579EmailStyle22
{font-family:sans-serif;color:#1F497D;}#yiv2571733579
.yiv2571733579MsoChpDefault {font-size:10.0pt;} _filtered #yiv2571733579
{margin:1.0in 1.0in 1.0in 1.0in;}#yiv2571733579 div.yiv2571733579WordSection1
{}#yiv2571733579
Are you using any rack aware topology? What are your partition keys? Is it
possible that your partition keys do not divide up as cleanly as you would like
across the cluster because the data is not evenly distributed (by partition
key)?
Sean Durity
lord of the (C*) rings (Staff Systems Engineer – Cassandra)
MTC 2250
#cassandra - for the latest news and updates
From: learner dba <[email protected]>
Sent: Monday, June 18, 2018 2:06 PM
To: User cassandra.apache.org <[email protected]>
Subject: [EXTERNAL] Cluster is unbalanced
Hi,
Data volume varies a lot in our two DC cluster:
Load Tokens Owns
20.01 GiB 256 ?
65.32 GiB 256 ?
60.09 GiB 256 ?
46.95 GiB 256 ?
50.73 GiB 256 ?
kaiprodv2
=========
/Leaving/Joining/Moving
Load Tokens Owns
25.19 GiB 256 ?
30.26 GiB 256 ?
9.82 GiB 256 ?
20.54 GiB 256 ?
9.7 GiB 256 ?
I ran clearsnapshot, garbagecollect and cleanup, but it increased the size on
heavier nodes instead of decreasing. Based on nodetool cfstats, I can see
partition keys on each node varies a lot:
Number of partitions (estimate): 3142552
Number of partitions (estimate): 15625442
Number of partitions (estimate): 15244021
Number of partitions (estimate): 9592992
Number of partitions (estimate): 15839280
How can I diagnose this imbalance further?
