As an update to this thread, we conducted several tests with
Cassandra-1.2.9, varying parameters such as partitioner
(Murmur3Partitioner/RandomParttioner), using NetworkToplogyStrategy (with
Ec2Snitch) / SimpleStrategy (with SimpleSnitch) across 2 Availability zones
and 1 AZ. We also tested the configurations separately with vnodes and
without vnodes.
Every time before each test, we wiped the cassandra cluster data and
commitlog folders and restarted with an empty cassandra db. However, in all
the cases using 1.2.9 we continued to see very heavy imbalance across the
nodes as reported in this thread.
We then tested the same exports with cassandra 1.2.5 version that we had
been testing previously (without vnodes across 2 AZs) and the data was
balanced across the nodes of the cluster. The output from bin/nodetool
status is attached.
Was there some change from 1.2.5 to 1.2.9 that could be responsible for the
imbalance or is there some parameter setting that we may have completely
missed in our configuration wrt 1.2.9? Has anyone else experienced such an
imbalance issue?
Also, we were contemplating on using vnodes with NetworkTopologyStrategy
(We want to replicate data across 2 AZs)
We came across the following links that mention that vnodes with
NetworkToplogyStrategy may create hotspots and the issue is marked as Open.
Does that mean using vnodes with NetworkToplogyStrategy is a bad idea?
[ https://issues.apache.org/jira/browse/CASSANDRA-4658 ,
https://issues.apache.org/jira/browse/CASSANDRA-3810 ,
https://issues.apache.org/jira/browse/CASSANDRA-4123 ] .
Thanks again for all your replies.
Suruchi
On Fri, Sep 20, 2013 at 7:04 PM, Robert Coli <[email protected]> wrote:
> On Fri, Sep 20, 2013 at 3:42 PM, Suruchi Deodhar <
> [email protected]> wrote:
>
>> Using the nodes in the same availability zone(us-east-1b), we still get a
>> highly imbalanced cluster. The nodetool status and ring output is attached.
>> Even after running repairs, the cluster does not seem to balance.
>>
>
> If your cluster doesn't experience exceptions when loading and/or store a
> lot of hints, repair is almost certainly just wasting your and your CPU's
> time.
>
> =Rob
>
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns Host ID
Token Rack
UN 10.151.86.146 362.52 MB 4.2% bcc08da4-58d8-4777-a838-253a997874db
-9223372036854775808 1a
UN 10.87.83.107 359.71 MB 4.2% 7af64cbc-c225-49a4-8750-8c4229042b91
-8454757700450211158 1b
UN 10.238.137.250 360.64 MB 4.2% dad2d2fb-784f-48c8-bca8-890bfa5181a2
-7686143364045646508 1a
UN 10.120.249.140 360.16 MB 4.2% 28826628-110b-4b46-8dec-8ad9b99a7459
-6917529027641081858 1b
UN 10.137.7.90 360.23 MB 4.2% c3379759-7c27-4ce7-87d1-eb63d69f6a30
-6148914691236517208 1a
UN 10.87.90.42 365.93 MB 4.2% cc23d1d5-e172-4d76-983b-0f4a7c410873
-5380300354831952558 1b
UN 10.238.133.174 364.67 MB 4.2% 7950da0a-84f3-49c0-ad4f-ad4860192320
-4611686018427387908 1a
UN 10.93.31.44 363.22 MB 4.2% 17016473-18c4-47be-942e-250216a6c0c4
-3843071682022823258 1b
UN 10.238.170.159 361.34 MB 4.2% 9c959ba6-888d-434f-8d60-3f8b5e87f933
-3074457345618258608 1a
UN 10.93.91.139 361.39 MB 4.2% d872c494-a9e5-452b-8789-8c30a27854f6
-2305843009213693958 1b
UN 10.137.20.183 363.02 MB 4.2% ce03bda4-587e-455c-8195-40115876793d
-1537228672809129308 1a
UN 10.87.75.147 362 MB 4.2% ef65f745-81cb-444a-8af9-53c860cfcca3
-768614336404564658 1b
UN 10.136.11.40 365.43 MB 4.2% 4d2cb28f-43d9-4e74-90d1-274344405ee8 -8
1a
UN 10.123.95.248 365.24 MB 4.2% f2d611d9-b275-4b79-9567-dbe44b0dc158
768614336404564642 1b
UN 10.151.49.88 363.49 MB 4.2% f1a02203-3064-4a63-9ace-dcc3dd9a3928
1537228672809129292 1a
UN 10.93.77.166 364.3 MB 4.2% acd22aa4-c0d2-42b5-85bd-023877f2787a
2305843009213693942 1b
UN 10.138.2.20 362.56 MB 4.2% 58abf545-bb4e-4a5c-81d1-7af0b2726704
3074457345618258592 1a
UN 10.90.246.128 362.63 MB 4.2% f654845c-0524-4279-abd9-e83b3458a2c8
3843071682022823242 1b
UN 10.138.10.9 362.88 MB 4.2% c6b086ef-4b75-4192-881a-a2dc85e8daa0
4611686018427387892 1a
UN 10.93.5.157 360.4 MB 4.2% 46a8cd4b-83dc-49d4-8e42-7cd55bebd69e
5380300354831952542 1b
UN 10.236.138.169 360.05 MB 4.2% 0c909de3-1c26-413b-baeb-a1d46d56e1bd
6148914691236517192 1a
UN 10.92.231.170 362.99 MB 4.2% ba9f767d-96c1-4596-b385-ae3694464289
6917529027641081842 1b
UN 10.238.133.97 365.93 MB 4.2% 23926ed9-bee6-4639-9d86-2de3829925ef
7686143364045646492 1a
UN 10.87.87.240 364.74 MB 4.2% abd52d1a-e462-4085-889e-5c224fb580dd
8454757700450211142 1b
