We also run into the same problem when trying to load data from a 8 node cluster (C* 1.2.1, Vnodes and same rack) into a 9 node cluster (C* 1.2.11, Vnodes and different racks) using sstableloader.
We observed that a Key that clearly belonged to 3 nodes (using nodetool getendpoints) has not been sent to any of them. Why? Best regards, Francisco. On Sep 24, 2013, at 7:12 PM, Suruchi Deodhar <suruchi.deod...@generalsentiment.com> wrote: > As an update to this thread, we conducted several tests with Cassandra-1.2.9, > varying parameters such as partitioner (Murmur3Partitioner/RandomParttioner), > using NetworkToplogyStrategy (with Ec2Snitch) / SimpleStrategy (with > SimpleSnitch) across 2 Availability zones and 1 AZ. We also tested the > configurations separately with vnodes and without vnodes. > > Every time before each test, we wiped the cassandra cluster data and > commitlog folders and restarted with an empty cassandra db. However, in all > the cases using 1.2.9 we continued to see very heavy imbalance across the > nodes as reported in this thread. > > We then tested the same exports with cassandra 1.2.5 version that we had been > testing previously (without vnodes across 2 AZs) and the data was balanced > across the nodes of the cluster. The output from bin/nodetool status is > attached. > > Was there some change from 1.2.5 to 1.2.9 that could be responsible for the > imbalance or is there some parameter setting that we may have completely > missed in our configuration wrt 1.2.9? Has anyone else experienced such an > imbalance issue? > > Also, we were contemplating on using vnodes with NetworkTopologyStrategy (We > want to replicate data across 2 AZs) > We came across the following links that mention that vnodes with > NetworkToplogyStrategy may create hotspots and the issue is marked as Open. > Does that mean using vnodes with NetworkToplogyStrategy is a bad idea? > > [ https://issues.apache.org/jira/browse/CASSANDRA-4658 , > https://issues.apache.org/jira/browse/CASSANDRA-3810 , > https://issues.apache.org/jira/browse/CASSANDRA-4123 ] . > > Thanks again for all your replies. > > Suruchi > > > > > > On Fri, Sep 20, 2013 at 7:04 PM, Robert Coli <rc...@eventbrite.com> wrote: > On Fri, Sep 20, 2013 at 3:42 PM, Suruchi Deodhar > <suruchi.deod...@generalsentiment.com> wrote: > Using the nodes in the same availability zone(us-east-1b), we still get a > highly imbalanced cluster. The nodetool status and ring output is attached. > Even after running repairs, the cluster does not seem to balance. > > If your cluster doesn't experience exceptions when loading and/or store a lot > of hints, repair is almost certainly just wasting your and your CPU's time. > > =Rob > > <status_1.2.5_wo_vnodes_lz4.txt>
