Good point Varun. Here is how I understand it but feel free to disagree.
According to the documentation the load column in the nodetool status output is "The amount of file system data under the cassandra data directory after excluding all content in the snapshots subdirectories. Because all SSTable data files are included, any data that is not cleaned up, such as TTL-expired cell or tombstoned data) is counted.". nodetool cleanup has been specifically added to remove unwanted data after adding a new node to the cluster. I am assuming its too expensive to calculate the actual load amount excluding unwanted data. Thus it seems the tool is working according to specification. Cheers, Akhil On Fri, Jun 16, 2017 at 4:39 PM, Varun Gupta <var...@uber.com> wrote: > > Akhil, > > As per the blog, nodetool status shows data size for node1 even for token > ranges it does not own. Ain't this is bug in Cassandra? > > Yes, on disk data will be present but it should be reflected in nodetool > status. > > On Thu, Jun 15, 2017 at 6:17 PM, Akhil Mehra <akhilme...@gmail.com> wrote: > >> Hi, >> >> I put together a blog explaining possible reasons for an unbalanced >> Cassandra nodes. >> >> http://abiasforaction.net/unbalanced-cassandra-cluster/ >> >> Let me know if you have any questions. >> >> Cheers, >> Akhil >> >> >> On Thu, Jun 15, 2017 at 5:54 PM, Affan Syed <as...@an10.io> wrote: >> >>> John, >>> >>> I am a co-worker with Junaid -- he is out sick, so just wanted to >>> confirm that one of your shots in the dark is correct. This is a RF of 1x >>> >>> "CREATE KEYSPACE orion WITH replication = {'class': 'SimpleStrategy', >>> 'replication_factor': '1'} AND durable_writes = true;" >>> >>> However, how does the RF affect the redistribution of key/data? >>> >>> Affan >>> >>> - Affan >>> >>> On Wed, Jun 14, 2017 at 1:16 AM, John Hughes <johnthug...@gmail.com> >>> wrote: >>> >>>> OP, I was just looking at your original numbers and I have some >>>> questions: >>>> >>>> 270GB on one node and 414KB on the other, but something close to 50/50 >>>> on "Owns(effective)". >>>> What replication factor are your keyspaces set up with? 1x or 2x or ?? >>>> >>>> I would say you are seeing 50/50 because the tokens are allocated >>>> 50/50(others on the list please correct what are for me really just >>>> assumptions!!!!), but I would hazard a guess that your replication factor >>>> is still 1x, so it isn't moving anything around. Or your keyspace >>>> rplication is incorrect and isn't being distributed(I have had issues with >>>> the AWSMultiRegionSnitch and not getting the region correct[us-east vs >>>> us-east-1). It doesn't throw an error, but it doesn't work very well either >>>> =) >>>> >>>> Can you do a 'describe keyspace XXX' and show the first line(the CREATE >>>> KEYSPACE line). >>>> >>>> Mind you, these are all just shots in the dark from here. >>>> >>>> Cheers, >>>> >>>> >>>> On Tue, Jun 13, 2017 at 3:13 AM Junaid Nasir <jna...@an10.io> wrote: >>>> >>>>> Is the OP expecting a perfect 50%/50% split? >>>>> >>>>> >>>>> best result I got was 240gb/30gb split, which I think is not properly >>>>> balanced. >>>>> >>>>> >>>>>> Also, what are your outputs when you call out specific keyspaces? Do >>>>>> the numbers get more even? >>>>> >>>>> >>>>> i don't know what you mean by *call out specific key spaces?* can you >>>>> please explain that a bit. >>>>> >>>>> >>>>> If your schema is not modelled correctly you can easily end up >>>>>> unevenly distributed data. >>>>> >>>>> >>>>> I think that is the problem. initial 270gb data might not by modeled >>>>> correctly. I have run a lot of tests on 270gb data including downsizing it >>>>> to 5gb, they all resulted in same uneven distribution. I also tested a >>>>> dummy dataset of 2gb which was balanced evenly. coming from rdb, I didn't >>>>> give much thought to data modeling. can anyone please point me to some >>>>> resources regarding this problem. >>>>> >>>>> On Tue, Jun 13, 2017 at 3:24 AM, Akhil Mehra <akhilme...@gmail.com> >>>>> wrote: >>>>> >>>>>> Great point John. >>>>>> >>>>>> The OP should also note that data distribution also depends on your >>>>>> schema and incoming data profile. >>>>>> >>>>>> If your schema is not modelled correctly you can easily end up >>>>>> unevenly distributed data. >>>>>> >>>>>> Cheers, >>>>>> Akhil >>>>>> >>>>>> On Tue, Jun 13, 2017 at 3:36 AM, John Hughes <johnthug...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Is the OP expecting a perfect 50%/50% split? That, to my experience, >>>>>>> is not going to happen, it is almost always shifted from a fraction of a >>>>>>> percent to a couple percent. >>>>>>> >>>>>>> Datacenter: eu-west >>>>>>> =================== >>>>>>> Status=Up/Down >>>>>>> |/ State=Normal/Leaving/Joining/Moving >>>>>>> -- Address Load Tokens Owns (effective) Host ID >>>>>>> Rack >>>>>>> UN XX.XX.XX.XX 22.71 GiB 256 47.6% >>>>>>> 57dafdde-2f62-467c-a8ff-c91e712f89c9 1c >>>>>>> UN XX.XX.XX.XX 17.17 GiB 256 51.3% >>>>>>> d2a65c51-087d-48de-ae1f-a41142eb148d 1b >>>>>>> UN XX.XX.XX.XX 26.15 GiB 256 52.4% >>>>>>> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c 1c >>>>>>> UN XX.XX.XX.XX 16.64 GiB 256 50.2% >>>>>>> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e 1a >>>>>>> UN XX.XX.XX.XX 24.39 GiB 256 49.8% >>>>>>> fd92525d-edf2-4974-8bc5-a350a8831dfa 1a >>>>>>> UN XX.XX.XX.XX 23.8 GiB 256 48.7% >>>>>>> bdc597c0-718c-4ef6-b3ef-7785110a9923 1b >>>>>>> >>>>>>> Though maybe part of what you are experiencing can be cleared up by >>>>>>> repair/compaction/cleanup. Also, what are your outputs when you call out >>>>>>> specific keyspaces? Do the numbers get more even? >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilme...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> auto_bootstrap is true by default. Ensure its set to true. On >>>>>>>> startup look at your logs for your auto_bootstrap value. Look at the >>>>>>>> node >>>>>>>> configuration line in your log file. >>>>>>>> >>>>>>>> Akhil >>>>>>>> >>>>>>>> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jna...@an10.io> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> No, I didn't set it (left it at default value) >>>>>>>>> >>>>>>>>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az1...@att.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Did you make sure auto_bootstrap property is indeed set to >>>>>>>>>> [true] when you added the node? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *From:* Junaid Nasir [mailto:jna...@an10.io] >>>>>>>>>> *Sent:* Monday, June 05, 2017 6:29 AM >>>>>>>>>> *To:* Akhil Mehra <akhilme...@gmail.com> >>>>>>>>>> *Cc:* Vladimir Yudovin <vla...@winguzone.com>; >>>>>>>>>> user@cassandra.apache.org >>>>>>>>>> *Subject:* Re: Convert single node C* to cluster (rebalancing >>>>>>>>>> problem) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> not evenly, i have setup a new cluster with subset of data >>>>>>>>>> (around 5gb). using the configuration above I am getting these >>>>>>>>>> results >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Datacenter: datacenter1 >>>>>>>>>> >>>>>>>>>> ======================= >>>>>>>>>> >>>>>>>>>> Status=Up/Down >>>>>>>>>> >>>>>>>>>> |/ State=Normal/Leaving/Joining/Moving >>>>>>>>>> >>>>>>>>>> -- Address Load Tokens Owns (effective) Host ID >>>>>>>>>> Rack >>>>>>>>>> >>>>>>>>>> UN 10.128.2.1 4.86 GiB 256 44.9% >>>>>>>>>> e4427611-c247-42ee-9404-371e177f5f17 rack1 >>>>>>>>>> >>>>>>>>>> UN 10.128.2.10 725.03 MiB 256 55.1% >>>>>>>>>> 690d5620-99d3-4ae3-aebe-8f33af54a08b rack1 >>>>>>>>>> >>>>>>>>>> is there anything else I can tweak/check to make the distribution >>>>>>>>>> even? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilme...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> So now the data is evenly balanced in both nodes? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Refer to the following documentation to get a better >>>>>>>>>> understanding of the roc_address and the broadcast_rpc_address >>>>>>>>>> https://www.instaclustr.com/demystifyi >>>>>>>>>> ng-cassandras-broadcast_address/ >>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Faddress_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=MaTA43pugg78xQNfaOQElhyvd8k7CjVqZPr3IWALdWI&e=>. >>>>>>>>>> I am surprised that your node started up with rpc_broadcast_address >>>>>>>>>> set as this is an unsupported property. I am assuming you are >>>>>>>>>> using Cassandra version 3.10. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Akhil >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2/06/2017, at 11:06 PM, Junaid Nasir <jna...@an10.io> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I am able to get it working. I added a new node with following >>>>>>>>>> changes >>>>>>>>>> >>>>>>>>>> #rpc_address:0.0.0.0 >>>>>>>>>> >>>>>>>>>> rpc_address: 10.128.1.11 >>>>>>>>>> >>>>>>>>>> #rpc_broadcast_address:10.128.1.11 >>>>>>>>>> >>>>>>>>>> rpc_address was set to 0.0.0.0, (I ran into a problem previously >>>>>>>>>> regarding remote connection and made these changes >>>>>>>>>> https://stackoverflow.com/questions/12236898/apache-cassandr >>>>>>>>>> a-remote-access >>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_12236898_apache-2Dcassandra-2Dremote-2Daccess&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=oj8BCLiyBDqqVQNqfGohFeujtqlzAkd-uwS878d4mg4&e=> >>>>>>>>>> ) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> should it be happening? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin < >>>>>>>>>> vla...@winguzone.com> wrote: >>>>>>>>>> >>>>>>>>>> Did you run "nodetool cleanup" on first node after second was >>>>>>>>>> bootstrapped? It should clean rows not belonging to node after tokens >>>>>>>>>> changed. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Best regards, Vladimir Yudovin, >>>>>>>>>> >>>>>>>>>> *Winguzone >>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com_-3Ffrom-3Dlist&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=Q1M5YRAsw0iUQKOIulEmO72RhdENQCRhpqZSjgxxHos&e=> >>>>>>>>>> - Cloud Cassandra Hosting* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ---- On Wed, 31 May 2017 03:55:54 -0400 *Junaid Nasir >>>>>>>>>> <jna...@an10.io <jna...@an10.io>>* wrote ---- >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Cassandra ensure that adding or removing nodes are very easy and >>>>>>>>>> that load is balanced between nodes when a change is made. but it's >>>>>>>>>> not >>>>>>>>>> working in my case. >>>>>>>>>> >>>>>>>>>> I have a single node C* deployment (with 270 GB of data) and want >>>>>>>>>> to load balance the data on multiple nodes, I followed this guide >>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_cassandra_2.1_cassandra_operations_ops-5Fadd-5Fnode-5Fto-5Fcluster-5Ft.html&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=xnLuX4qqSZxVuY6Gz2NXRdc6TG8J7at9kDhkhBnWpnM&e=> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> `nodetool status` shows 2 nodes but load is not balanced between >>>>>>>>>> them >>>>>>>>>> >>>>>>>>>> Datacenter: dc1 >>>>>>>>>> >>>>>>>>>> =============== >>>>>>>>>> >>>>>>>>>> Status=Up/Down >>>>>>>>>> >>>>>>>>>> |/ State=Normal/Leaving/Joining/Moving >>>>>>>>>> >>>>>>>>>> -- Address Load Tokens Owns (effective) Host ID >>>>>>>>>> Rack >>>>>>>>>> >>>>>>>>>> UN 10.128.0.7 270.75 GiB 256 48.6% >>>>>>>>>> 1a3f6faa-4376-45a8-9c20-11480ae5664c rack1 >>>>>>>>>> >>>>>>>>>> UN 10.128.0.14 414.36 KiB 256 51.4% >>>>>>>>>> 66a89fbf-08ba-4b5d-9f10-55d52a199b41 rack1 >>>>>>>>>> >>>>>>>>>> I also ran 'nodetool repair' on new node but result is same. any >>>>>>>>>> pointers would be appreciated :) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> conf file of new node >>>>>>>>>> >>>>>>>>>> cluster_name: 'cluster1' >>>>>>>>>> >>>>>>>>>> - seeds: "10.128.0.7" >>>>>>>>>> num_tokens: 256 >>>>>>>>>> >>>>>>>>>> endpoint_snitch: GossipingPropertyFileSnitch >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Junaid >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> >> >
