Did you start out your cluster after wiping all the sstables and commit logs?
On Fri, Sep 20, 2013 at 3:42 PM, Suruchi Deodhar < suruchi.deod...@generalsentiment.com> wrote: > We have been trying to resolve this issue to find a stable configuration > that can give us a balanced cluster with equally distributed data across > the nodes. > > We tried running a smaller 12 node cluster with following parameters: > > placement_strategy = NetworkTopologyStrategy ( 12 nodes in us-east-1b ) > partitioner=Murmur3Partitioner > compaction strategy = LeveledCompactionStrategy > replication factor = 2 > snitch = EC2Snitch > vnodes with num_tokens = 256 ( 256 tokens per node ) > > Using the nodes in the same availability zone(us-east-1b), we still get a > highly imbalanced cluster. The nodetool status and ring output is attached. > Even after running repairs, the cluster does not seem to balance. > > Datacenter: us-east >> =================== >> Status=Up/Down >> |/ State=Normal/Leaving/Joining/Moving >> -- Address Load Tokens Owns (effective) Host >> ID Rack >> UN 10.120.249.140 641.11 KB 256 16.7% >> 90320ebc-cd03-4303-9f91-a8eb534ecaa4 1b >> UN 10.90.246.128 1.04 MB 256 16.4% >> 14352484-07c2-4cf0-9c12-e53d9e19f23b 1b >> UN 10.123.95.248 4.21 GB 256 17.7% >> 3d520c2a-f9a0-480a-83d8-675ed3f46c7d 1b >> UN 10.87.87.240 337.46 KB 256 16.6% >> 11bfebe6-f40c-4d4e-a76e-f6287a27fa53 1b >> UN 10.93.5.157 4.21 GB 256 17.0% >> 5a20ca7b-37fa-4e5b-86d4-c47c80e3d803 1b >> UN 10.92.231.170 1021.72 KB 256 15.5% >> e4835262-e27c-4da3-b45d-758a860f506e 1b >> UN 10.93.31.44 653.58 KB 256 16.5% >> 19d24eda-b9bf-42cc-8463-5be71d28da9d 1b >> UN 10.93.91.139 298.81 KB 256 17.8% >> 6d8d5879-4340-4d7c-960d-a12a80a9fac3 1b >> UN 10.87.90.42 3.75 MB 256 16.8% >> ff64cfc0-d614-48b9-8e96-aee99114c6ff 1b >> UN 10.87.75.147 3.55 MB 256 16.6% >> ec1f182d-bb01-4546-a523-2a42df158ee0 1b >> UN 10.93.77.166 333.88 KB 256 16.0% >> 0964902c-736d-411a-bed5-341dbc96813f 1b >> UN 10.87.83.107 543.2 KB 256 16.4% >> 924fa50c-d253-41e8-b2fb-d1be2942f2c8 1b >> >> After repairs- status >> > > Datacenter: us-east > =================== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host > ID Rack > UN 10.120.249.140 82.26 MB 256 16.7% > 90320ebc-cd03-4303-9f91-a8eb534ecaa4 1b > UN 10.90.246.128 97.61 MB 256 16.4% > 14352484-07c2-4cf0-9c12-e53d9e19f23b 1b > UN 10.123.95.248 4.25 GB 256 17.7% > 3d520c2a-f9a0-480a-83d8-675ed3f46c7d 1b > UN 10.87.87.240 95.05 MB 256 16.6% > 11bfebe6-f40c-4d4e-a76e-f6287a27fa53 1b > UN 10.93.5.157 4.25 GB 256 17.0% > 5a20ca7b-37fa-4e5b-86d4-c47c80e3d803 1b > UN 10.92.231.170 86.54 MB 256 15.5% > e4835262-e27c-4da3-b45d-758a860f506e 1b > UN 10.93.31.44 102.42 MB 256 16.5% > 19d24eda-b9bf-42cc-8463-5be71d28da9d 1b > UN 10.93.91.139 82.85 MB 256 17.8% > 6d8d5879-4340-4d7c-960d-a12a80a9fac3 1b > UN 10.87.90.42 134.76 MB 256 16.8% > ff64cfc0-d614-48b9-8e96-aee99114c6ff 1b > UN 10.87.75.147 101.15 MB 256 16.6% > ec1f182d-bb01-4546-a523-2a42df158ee0 1b > UN 10.93.77.166 73.66 MB 256 16.0% > 0964902c-736d-411a-bed5-341dbc96813f 1b > UN 10.87.83.107 93.73 MB 256 16.4% > 924fa50c-d253-41e8-b2fb-d1be2942f2c8 1b > > > So essentially, using NetworkTopology strategy even with a single AZ is > not giving us a balanced cluster. Is there any alternative that we can try > for a stable cluster? We want to use vnodes. > > Thanks, > Suruchi > > > On Fri, Sep 20, 2013 at 1:31 PM, Robert Coli <rc...@eventbrite.com> wrote: > >> On Fri, Sep 20, 2013 at 9:24 AM, Jayadev Jayaraman >> <jdisal...@gmail.com>wrote: >> >>> As a follow-up, is operating a Cassandra cluster with machines on >>> multiple racks and vnodes bound to cause load imbalance ? Shouldn't >>> token-ranges assigned to individual machines via their vnodes be >>> approximately balanced ? We're otherwise unable to explain why this >>> imbalance occurs. ( it shouldn't be the fault of the Murmur3 partitioner >>> which guarantees a uniform distribution of keys across token-ranges >>> according to the doc. ) >>> >> >> https://issues.apache.org/jira/browse/CASSANDRA-4658 >> https://issues.apache.org/jira/browse/CASSANDRA-4123 >> >> tl;dr : Yes, vnodes makes the imbalance factor slightly greater, but in >> practice it "shouldn't" be significant. >> >> If you are experiencing significant imbalance, one or both of those >> tickets may be an appropriate forum to describe your experience. >> >> =Rob >> >> > >
