Like I said in my previous reply that I am not sure if that is the problem and that's why I thought it would be a good test to do your test with cluster in one RACK only.
I'll take a look at your ring output today. Did you also post cfstats output? On Fri, Sep 20, 2013 at 9:24 AM, Jayadev Jayaraman <jdisal...@gmail.com>wrote: > As a follow-up, is operating a Cassandra cluster with machines on multiple > racks and vnodes bound to cause load imbalance ? Shouldn't token-ranges > assigned to individual machines via their vnodes be approximately balanced > ? We're otherwise unable to explain why this imbalance occurs. ( it > shouldn't be the fault of the Murmur3 partitioner which guarantees a > uniform distribution of keys across token-ranges according to the doc. ) > > > On Thu, Sep 19, 2013 at 8:56 PM, Jayadev Jayaraman <jdisal...@gmail.com>wrote: > >> We're using NetworkTopologyStrategy for our placement_strategy and our >> primary motivation for placing nodes on more than 1 rack is to prevent >> showstoppers caused by localized faults leading to multiple EC2 instances >> in the same availability zone (rack) becoming unreachable (which we have >> encountered before). >> >> I've attached the output of "nodetool ring" here. >> >> >> On Thu, Sep 19, 2013 at 8:35 PM, Mohit Anchlia <mohitanch...@gmail.com>wrote: >> >>> Other thing I noticed is that you are using mutiple RACKS and that might >>> be contributing factor to it. However, I am not sure. >>> >>> Can you paste the output of nodetool cfstats and ring? Is it possible to >>> run the same test but keeping all the nodes in one rack? >>> >>> I think you should open a JIRA if you are able to reproduce this. >>> >>> On Thu, Sep 19, 2013 at 4:41 PM, Jayadev Jayaraman >>> <jdisal...@gmail.com>wrote: >>> >>>> We ran nodetool repair on all nodes for all Keyspaces / CFs, restarted >>>> cassandra and this is what we get for nodetool status : >>>> >>>> bin/nodetool -h localhost status >>>> Datacenter: us-east >>>> =================== >>>> Status=Up/Down >>>> |/ State=Normal/Leaving/Joining/Moving >>>> -- Address Load Tokens Owns (effective) Host ID >>>> Rack >>>> UN 10.238.133.174 885.36 MB 256 8.4% >>>> e41d8863-ce37-4d5c-a428-bfacea432a35 1a >>>> UN 10.238.133.97 468.66 MB 256 7.7% >>>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a >>>> UN 10.151.86.146 1.08 GB 256 8.0% >>>> 8952645d-4a27-4670-afb2-65061c205734 1a >>>> UN 10.138.10.9 941.44 MB 256 8.6% >>>> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a >>>> UN 10.87.87.240 99.69 MB 256 8.6% >>>> ea066827-83bc-458c-83e8-bd15b7fc783c 1b >>>> UN 10.93.5.157 87.44 MB 256 7.6% >>>> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b >>>> UN 10.238.137.250 561.42 MB 256 7.8% >>>> 84301648-afff-4f06-aa0b-4be421e0d08f 1a >>>> UN 10.92.231.170 893.75 MB 256 9.3% >>>> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b >>>> UN 10.138.2.20 31.89 MB 256 7.9% >>>> a6d4672a-0915-4c64-ba47-9f190abbf951 1a >>>> UN 10.93.31.44 312.52 MB 256 7.8% >>>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b >>>> UN 10.93.91.139 30.46 MB 256 8.1% >>>> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b >>>> UN 10.236.138.169 260.15 MB 256 9.1% >>>> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a >>>> UN 10.137.7.90 38.45 MB 256 7.4% >>>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a >>>> UN 10.93.77.166 867.15 MB 256 8.8% >>>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b >>>> UN 10.120.249.140 863.98 MB 256 9.4% >>>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b >>>> UN 10.90.246.128 242.63 MB 256 8.4% >>>> 054911ec-969d-43d9-aea1-db445706e4d2 1b >>>> UN 10.123.95.248 171.51 MB 256 7.2% >>>> a17deca1-9644-4520-9e62-ac66fc6fef60 1b >>>> UN 10.136.11.40 33.8 MB 256 8.5% >>>> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a >>>> UN 10.87.90.42 38.01 MB 256 8.0% >>>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b >>>> UN 10.87.75.147 579.29 MB 256 8.3% >>>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b >>>> UN 10.151.49.88 151.06 MB 256 8.9% >>>> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a >>>> UN 10.87.83.107 512.91 MB 256 8.3% >>>> 0019439b-9f8a-4965-91b8-7108bbb55593 1b >>>> UN 10.238.170.159 85.04 MB 256 9.4% >>>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a >>>> UN 10.137.20.183 167.41 MB 256 8.4% >>>> 15951592-8ab2-473d-920a-da6e9d99507d 1a >>>> >>>> It doesn't seem to have changed by much. The loads are still highly >>>> uneven. >>>> >>>> As for the number of keys in each node's CFs : the largest node now >>>> has 5589120 keys for the column-family that had 6527744 keys before >>>> (load is now 1.08 GB as compares to 1.05 GB before), while the smallest >>>> node now has 71808 keys as compared to 3840 keys before (load is now >>>> 31.89 MB as compares to 1.12 MB before). >>>> >>>> >>>> On Thu, Sep 19, 2013 at 5:18 PM, Mohit Anchlia >>>> <mohitanch...@gmail.com>wrote: >>>> >>>>> Can you run nodetool repair on all the nodes first and look at the >>>>> keys? >>>>> >>>>> >>>>> On Thu, Sep 19, 2013 at 1:22 PM, Suruchi Deodhar < >>>>> suruchi.deod...@generalsentiment.com> wrote: >>>>> >>>>>> Yes, the key distribution does vary across the nodes. For example, on >>>>>> the node with the highest data, Number of Keys (estimate) is 6527744 for >>>>>> a >>>>>> particular column family, whereas for the same column family on the node >>>>>> with least data, Number of Keys (estimate) = 3840. >>>>>> >>>>>> Is there a way to control this distribution by setting some parameter >>>>>> of cassandra. >>>>>> >>>>>> I am using the Murmur3 partitioner with NetworkTopologyStrategy. >>>>>> >>>>>> Thanks, >>>>>> Suruchi >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Sep 19, 2013 at 3:59 PM, Mohit Anchlia < >>>>>> mohitanch...@gmail.com> wrote: >>>>>> >>>>>>> Can you check cfstats to see number of keys per node? >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 19, 2013 at 12:36 PM, Suruchi Deodhar < >>>>>>> suruchi.deod...@generalsentiment.com> wrote: >>>>>>> >>>>>>>> Thanks for your replies. I wiped out my data from the cluster and >>>>>>>> also cleared the commitlog before restarting it with num_tokens=256. I >>>>>>>> then >>>>>>>> uploaded data using sstableloader. >>>>>>>> >>>>>>>> However, I am still not able to see a uniform distribution of data >>>>>>>> across nodes of the clusters. >>>>>>>> >>>>>>>> The output of the bin/nodetool -h localhost status commands looks >>>>>>>> like follows. Some nodes have data as low as 1.12MB while some have as >>>>>>>> high >>>>>>>> as 912.57 MB. >>>>>>>> >>>>>>>> Datacenter: us-east >>>>>>>> =================== >>>>>>>> Status=Up/Down >>>>>>>> |/ State=Normal/Leaving/Joining/Moving >>>>>>>> -- Address Load Tokens Owns (effective) Host >>>>>>>> ID Rack >>>>>>>> UN 10.238.133.174 856.66 MB 256 8.4% >>>>>>>> e41d8863-ce37-4d5c-a428-bfacea432a35 1a >>>>>>>> UN 10.238.133.97 439.02 MB 256 7.7% >>>>>>>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547 1a >>>>>>>> UN 10.151.86.146 1.05 GB 256 8.0% >>>>>>>> 8952645d-4a27-4670-afb2-65061c205734 1a >>>>>>>> UN 10.138.10.9 912.57 MB 256 8.6% >>>>>>>> 25ccea82-49d2-43d9-830c-b9c9cee026ec 1a >>>>>>>> UN 10.87.87.240 70.85 MB 256 8.6% >>>>>>>> ea066827-83bc-458c-83e8-bd15b7fc783c 1b >>>>>>>> UN 10.93.5.157 60.56 MB 256 7.6% >>>>>>>> 4ab9111c-39b4-4d15-9401-359d9d853c16 1b >>>>>>>> UN 10.92.231.170 866.73 MB 256 9.3% >>>>>>>> a18ce761-88a0-4407-bbd1-c867c4fecd1f 1b >>>>>>>> UN 10.238.137.250 533.77 MB 256 7.8% >>>>>>>> 84301648-afff-4f06-aa0b-4be421e0d08f 1a >>>>>>>> UN 10.93.91.139 478.45 KB 256 8.1% >>>>>>>> 682dd848-7c7f-4ddb-a960-119cf6491aa1 1b >>>>>>>> UN 10.138.2.20 1.12 MB 256 7.9% >>>>>>>> a6d4672a-0915-4c64-ba47-9f190abbf951 1a >>>>>>>> UN 10.93.31.44 282.65 MB 256 7.8% >>>>>>>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf 1b >>>>>>>> UN 10.236.138.169 223.66 MB 256 9.1% >>>>>>>> cbbf27b0-b53a-4530-bfdf-3764730b89d8 1a >>>>>>>> UN 10.137.7.90 11.36 MB 256 7.4% >>>>>>>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4 1a >>>>>>>> UN 10.93.77.166 837.64 MB 256 8.8% >>>>>>>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb 1b >>>>>>>> UN 10.120.249.140 838.59 MB 256 9.4% >>>>>>>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a 1b >>>>>>>> UN 10.90.246.128 216.75 MB 256 8.4% >>>>>>>> 054911ec-969d-43d9-aea1-db445706e4d2 1b >>>>>>>> UN 10.123.95.248 147.1 MB 256 7.2% >>>>>>>> a17deca1-9644-4520-9e62-ac66fc6fef60 1b >>>>>>>> UN 10.136.11.40 4.24 MB 256 8.5% >>>>>>>> 66be1173-b822-40b5-b650-cb38ae3c7a51 1a >>>>>>>> UN 10.87.90.42 11.56 MB 256 8.0% >>>>>>>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1 1b >>>>>>>> UN 10.87.75.147 549 MB 256 8.3% >>>>>>>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8 1b >>>>>>>> UN 10.151.49.88 119.86 MB 256 8.9% >>>>>>>> 57043573-ab1b-4e3c-8044-58376f7ce08f 1a >>>>>>>> UN 10.87.83.107 484.3 MB 256 8.3% >>>>>>>> 0019439b-9f8a-4965-91b8-7108bbb55593 1b >>>>>>>> UN 10.137.20.183 137.67 MB 256 8.4% >>>>>>>> 15951592-8ab2-473d-920a-da6e9d99507d 1a >>>>>>>> UN 10.238.170.159 49.17 MB 256 9.4% >>>>>>>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684 1a >>>>>>>> >>>>>>>> Is there something else that I should be doing differently? >>>>>>>> >>>>>>>> Thanks for your help! >>>>>>>> >>>>>>>> Suruchi >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 19, 2013 at 3:20 PM, Richard Low >>>>>>>> <rich...@wentnet.com>wrote: >>>>>>>> >>>>>>>>> The only thing you need to guarantee is that Cassandra doesn't >>>>>>>>> start with num_tokens=1 (the default in 1.2.x) or, if it does, that >>>>>>>>> you >>>>>>>>> wipe all the data before starting it with higher num_tokens. >>>>>>>>> >>>>>>>>> >>>>>>>>> On 19 September 2013 19:07, Robert Coli <rc...@eventbrite.com>wrote: >>>>>>>>> >>>>>>>>>> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar < >>>>>>>>>> suruchi.deod...@generalsentiment.com> wrote: >>>>>>>>>> >>>>>>>>>>> Do you suggest I should try with some other installation >>>>>>>>>>> mechanism? Are there any known problems with the tar installation of >>>>>>>>>>> cassandra 1.2.9 that I should be aware of? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I was asking in the context of this JIRA : >>>>>>>>>> >>>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-2356 >>>>>>>>>> >>>>>>>>>> Which does not seem to apply in your case! >>>>>>>>>> >>>>>>>>>> =Rob >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
