When you bootstrapped the node for the first time did you see log similar to the following: INFO [main] 2017-06-01 07:19:45,199 StorageService.java:1435 - JOINING: waiting for schema information to complete INFO [main] 2017-06-01 07:19:45,250 StorageService.java:1435 - JOINING: schema complete, ready to bootstrap INFO [main] 2017-06-01 07:19:45,251 StorageService.java:1435 - JOINING: waiting for pending range calculation INFO [main] 2017-06-01 07:19:45,251 StorageService.java:1435 - JOINING: calculation complete, ready to bootstrap INFO [main] 2017-06-01 07:19:45,251 StorageService.java:1435 - JOINING: getting bootstrap token INFO [main] 2017-06-01 07:19:45,341 StorageService.java:1435 - JOINING: sleeping 30000 ms for pending range setup INFO [main] 2017-06-01 07:20:15,342 StorageService.java:1435 - JOINING: Starting to bootstrap... INFO [main] 2017-06-01 07:20:15,562 StreamResultFuture.java:90 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae] Executing streaming plan for Bootstrap INFO [StreamConnectionEstablisher:1] 2017-06-01 07:20:15,568 StreamSession.java:266 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae] Starting streaming to /172.20.0.3 INFO [StreamConnectionEstablisher:1] 2017-06-01 07:20:15,591 StreamCoordinator.java:264 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae, ID#0] Beginning stream session with /172.20.0.3 INFO [STREAM-IN-/172.20.0.3:7000] 2017-06-01 07:20:16,369 StreamResultFuture.java:173 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae ID#0] Prepare completed. Receiving 4 files(4.046MiB), sending 0 files(0.000KiB) INFO [StreamReceiveTask:1] 2017-06-01 07:20:32,489 StreamResultFuture.java:187 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae] Session with /172.20.0.3 is complete INFO [StreamReceiveTask:1] 2017-06-01 07:20:32,529 StreamResultFuture.java:219 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae] All sessions completed INFO [StreamReceiveTask:1] 2017-06-01 07:20:32,535 StorageService.java:1491 - Bootstrap completed! for the tokens [2170857698622202367, 8517072504425343717, -3254771037524187900, -1597835042001935502, 6878847904605480741, 6816916215341820068, 6291189887494617640, -6855333019196580358, -6353317035112065873, 8838974905234547016, 8539810544447438397, -2357950949959511387, 1242077960532340887, 2914039668080386735, 3548015300105653368, 8973388453035242795, -2325235809399362967, -7078812010537656277, 768585495224455336, 7153512700965912517, 8625819392009074153, -6138302849441936958, -2594051958993427953, -735827743339795655, -8202727571538912843, 2180751358288507888, -7872842094207074012, -2926504780761300623, -3197260822146229664, 3411052656191450941, -9049284186987733291, -157882351668930258, 454637839762305232, -2305675997627138050, 5785282040753174988, 8604531769609599767, 4363117061247143957, -7255854383313210529, -3497611663121502480, -6788457421774336480, -7809767930173770420, 6591540654522244365, 1773283733607350132, 1347769111173669066, -7242556233623424655, -1552552727731631642, -1226243976028310059, -8221762326275074149, -7963893314043006091, -850542197910474448, 4219437099703910566, -8039365343972054221, 7756456412568178996, 4057327843751741693, 7155628666873897485, -483058846775660782, 6968839681845709305, 6396337738827005745, -5285173481531605912, 7254663657455123842, 871654822989271789, -604574593420741277, -2446461444470484127, -3707613591745746278, -26727542030118959, -7190990795521107837, 5388348291571480415, 4249499356533972018, 82469082189512791, -6389351372873749061, 5138413916027470955, 2542233707258091740, -4057927973990056143, 552933169018893618, -8237860380097407047, 6917383508758068288, 543382311932406672, -5671560690999322491, -1240369858424929757, 7394536427227616773, 4716882285905136652, 8260705434779371419, 3259812719139852593, -73864539388331289, -3573980475038135246, -1047139059901238511, -1734886021153324482, 8674873751672827600, 3564384074427511950, 2754071903665103098, -1230493021099846761, -2731315467436512731, -7845984767828231726, -8082165594257396645, -2298177264815779081, -3645421111048544165, 9142633389925493379, 7206663288804675578, 2305939212045070856, -5101738026249032246, 6268847697773786891, 5903922100677671597, -2001787466557152206, 1318502870562311928, 5784020265166141829, 5385229217299505171, 6010414616247875068, -8080602674779008196, -9189764569651551963, -8969124116887255329, -9040482343274988119, -8575947267671214955, -1786409930636352174, -757203989676123224, -6640569567328853730, 8431839804447545665, 6781635966829972979, -8328382509754233304, -3181089993114819214, 3243262023331941781, 4213737472390389773, -4046361821170607634, 8877904009116429296, -6931048276693039052, 4838006612846181604, -5561480934050473057, -470112649587309682, 3175935810873308999, -1693695808908080717, -3753035103371291265, -2607412695999984337, -8454963020263227780, 2037428931895594762, 1158209127301347406, -8092787384269386871, -7741092217712244823, 3213269181965324853, 3972662756438857798, -1808499161350392500, -5552429155141285488, 2768019514490102470, -2381885168558935166, 7598271141891576988, -5968675860637356104, -2178161882622813874, -2782395662355757709, 4662660828871465894, 6726990970215445064, 691223925843765893, -3732536320705038428, 2169053732177722520, -3467691490997179851, -3201672755574011994, -706634586120752453, -7297234535099792750, 4195063085031070570, 1024797669903232596, -5102042883065498245, -4295412307398568491, -686079656172478689, -7652228004418103329, -5922734755429174917, 6562442130946482224, 3419893918407781185, -7840156446781283061, -3209525913297552052, -8254134338430272746, -7543559272928655856, -6413145215334169356, 8189387753488304279, 133576451402117013, 6859840908124654784, -480832477584575919, -4466949465409090307, 2224334850433074431, -1077941859365184025, 7694746877711316656, -1541425238506019058, -2694798376156556512, -752352477219592169, 1911593773128947549, 1053380063512932771, 2369074212175473237, 1511764544820953277, -916955813829019462, -3389255868702958135, -4853221732440365560, 436528405098383237, 1482375829041075317, -5200032867873743382, -2359116065936986284, 2323558012249686328, -8714456372962073495, 8264655970455461946, -6377550880012897671, 1904431293553435268, 4281876827769946039, 129146370702108712, -7734952979230886745, -8242033642929677698, 2814257954427384288, -3054904268940900861, -450367636631970004, 8109127225549270088, 8020800599127010921, -2679593348517582224, -1752859515012359396, 4479567094782500113, 6631436750259638425, -5598101192868431054, 2794504876633839794, -1756288804973602742, -534202438069841833, 7027493063976551471, 3982737257386418900, -930887821667260064, 273533908252754639, -1440739443874625754, -2507143240715618713, 2102474087594804091, -7697404848019572605, -8658346826860002424, -4596011304911479578, 6459167476271897881, 6805481881299404326, -7080476038981754572, 5767237378480651194, -978605465701070447, -2322705636227730109, -681083377956364436, -5892502600903861513, -3301298109594166057, 8752787692307586763, -7189731757057511702, 8976826014517372173, -6975924228178934185, 4829963829825594588, -2886015764818641366, -7753659165428942667, -5923615337534732994, 7275900434506691605, 4164818915231369384, 5154263935677458810, 301044058442370317, -7764756071469383634, 7327453618420416121, -2412303784295409238, 781599758351322350, 6625371277047208299, -7611076121968222555, 6127681154442724891, 7313542022544470677, 7222174446262831041, -9138442343184845349, -4110447459792797189, 2534623752979121051] INFO [main] 2017-06-01 07:20:32,544 StorageService.java:1435 - JOINING: Finish joining ring
Regards, Akhil > On 1/06/2017, at 5:28 PM, Junaid Nasir <jna...@an10.io> wrote: > > sorry that was an old notetool status output (old cluster with a few > configuration changes). new configuration is > > -- Address Load Tokens Owns (effective) Host ID > Rack > UN 10.128.1.2 270.75 GiB 256 52.6% > e4427611-c247-42ee-9404-371e177f5f17 rack1 > UN 10.128.1.10 388.52 KiB 256 47.4% > 7059e622-7861-495b-a69f-5a68876364dd rack1 > > 10.128.1.2 is seed node, and no other C* node is running on the network. > data_file_directories on seed node is not in default location, it's on > another disk. if that helps > > On Thu, Jun 1, 2017 at 4:49 AM, Akhil Mehra <akhilme...@gmail.com > <mailto:akhilme...@gmail.com>> wrote: > Hi Junaid, > > I noticed in the log files that data is being streamed from/to 10.128.1.2 but > the address of your original node is 10.128.0.7. > > Are there any other Cassandra nodes on your local network the same cluster > name. > > Regards, > Akhil > > > 10.128.1.2 > > On 31/05/2017, at 10:15 PM, Junaid Nasir <jna...@an10.io > > <mailto:jna...@an10.io>> wrote: > > > > after nodetool repair on new node following log appears in cassandra log > > > > INFO [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496 > > StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] > > Starting streaming to /10.128.1.2 <http://10.128.1.2/> > > INFO [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501 > > StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac, > > ID#0] Beginning stream session with /10.128.1.2 <http://10.128.1.2/> > > INFO [STREAM-IN-/10.128.1.2:7000 <http://10.128.1.2:7000/>] 2017-05-31 > > 10:07:23,545 StreamResultFuture.java:173 - [Stream > > #f0eb9670-45e8-11e7-a17b-81e9a18c6eac ID#0] Prepare completed. Receiving 2 > > files(131.594KiB), sending 0 files(0.000KiB) > > INFO [StreamReceiveTask:1] 2017-05-31 10:07:23,705 > > StreamResultFuture.java:187 - [Stream > > #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] Session with /10.128.1.2 > > <http://10.128.1.2/> is complete > > INFO [StreamReceiveTask:1] 2017-05-31 10:07:23,707 > > StreamResultFuture.java:219 - [Stream > > #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] All sessions completed > > > > > > > > On Wed, May 31, 2017 at 2:50 PM, Junaid Nasir <jna...@an10.io > > <mailto:jna...@an10.io>> wrote: > > Cassandra version is 3.10, and yes its not a production server. i have seen > > some warnings in logs saying token exist on both servers. other than that > > nothing. if you need any more settings/details please ask. thank you for > > your time > > > > > > On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin > > <oleksandr.shul...@zalando.de <mailto:oleksandr.shul...@zalando.de>> wrote: > > On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir <jna...@an10.io > > <mailto:jna...@an10.io>> wrote: > > Cassandra ensure that adding or removing nodes are very easy and that load > > is balanced between nodes when a change is made. but it's not working in my > > case. > > I have a single node C* deployment (with 270 GB of data) and want to load > > balance the data on multiple nodes, > > > > I guess it's fair to assume this is not a production "cluster"? > > > > I followed this guide > > `nodetool status` shows 2 nodes but load is not balanced between them > > Datacenter: dc1 > > =============== > > Status=Up/Down > > |/ State=Normal/Leaving/Joining/M > > oving > > -- Address Load Tokens Owns (effective) Host ID Rack > > UN 10.128.0.7 270.75 GiB 256 48.6% > > 1a3f6faa-4376-45a8-9c20-11480a > > e5664c rack1 > > UN 10.128.0.14 414.36 KiB 256 51.4% > > 66a89fbf-08ba-4b5d-9f10-55d52a > > 199b41 rack1 > > I can imagine you are referring to Load column here, which shows only 400 > > KB for the new node. Did the newly added node actually bootstrap? Are > > there any error/warning messages in the Cassandra log? > > > > I also ran 'nodetool repair' on new node but result is same. any pointers > > would be appreciated :) > > > > Hm, this is not expected. Even if the node didn't bootstrap, repair should > > have streamed the data it is responsible for. > > > > conf file of new node > > cluster_name: 'cluster1' > > - seeds: "10.128.0.7" > > > > num_tokens: 256 > > endpoint_snitch: GossipingPropertyFileSnitch > > > > I see that you've tried to add the new node as a normal (non-seed) node, > > which supposed to work and it should have bootstrapped itself. Which > > version of Cassandra is it exactly? > > > > -- > > Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176 > > 127-59-707 > > > > > > > >
