Yes, that was it. Thanks a lot !! From: Joel Knighton [mailto:joel.knigh...@datastax.com] Sent: Thursday, April 7, 2016 10:02 AM To: user@cassandra.apache.org Subject: Re: Removing a DC
This sounds most like https://issues.apache.org/jira/browse/CASSANDRA-10371<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fCASSANDRA-10371&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7c82fcf2ffa417448f9e3d08d35f06513b%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=lOzyf0ZTq0zSuPoAHGj0S89SZKFhkfOBiTMqXPkRUjA%3d>. Are you on a version that could be affected by this issue? Best, Joel On Thu, Apr 7, 2016 at 11:51 AM, Anubhav Kale <anubhav.k...@microsoft.com<mailto:anubhav.k...@microsoft.com>> wrote: Hello, We removed a DC using instructions from https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_decomission_dc_t.html<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fdocs.datastax.com%2fen%2fcassandra%2f2.1%2fcassandra%2foperations%2fops_decomission_dc_t.html&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7c82fcf2ffa417448f9e3d08d35f06513b%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=KHIaLe3XffbhfbI1gdtaWVXkz3XlFdYqMgBeX33r1xs%3d> After all nodes were gone, 1. System.peers don’t have an entry for the nodes that were removed. (confirmed via a cqlsh query with consistency all) 2. Nodetool describecluster don’t show them 3. Nodetool gossipinfo show them as “LEFT”. However, logs continue to spew below and restarting the node doesn’t get rid of this. I am thinking a rolling restart of all nodes might fix it, but I am curious as to where is this information still held ? I don’t think this is causing any badness to the cluster, but I would like to get rid of this if possible. INFO [GossipStage:83] 2016-04-07 16:38:07,859 Gossiper.java:998 - InetAddress /10.1.200.14<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2f10.1.200.14&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7c82fcf2ffa417448f9e3d08d35f06513b%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=KFHbmVf%2fHO4VvHXBhCojmu4tcPXhmcXKYcqVSdPvFqY%3d> is now DOWN INFO [GossipStage:83] 2016-04-07 16:38:07,861 StorageService.java:1914 - Removing tokens[BLAH] for /10.1.200.14<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2f10.1.200.14&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7c82fcf2ffa417448f9e3d08d35f06513b%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=KFHbmVf%2fHO4VvHXBhCojmu4tcPXhmcXKYcqVSdPvFqY%3d> Thanks ! -- [https://lh6.googleusercontent.com/zujWCxiKGPoNTpU9rYwVo1GIvn0MXxFMiHJMgB_UZdESbDD0cn0gKnUw4UcYkyjI3uEbnxwVcjrGuvnXaTR4FoTt3F6u2whn_5qLRoExWTqSMYpv-9OtXIysw9rN5mXz564v5oY]<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.datastax.com%2f&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7c82fcf2ffa417448f9e3d08d35f06513b%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=nJkQYip0cW3aWnSZgPmGi4XQLeDVYs%2b7znVlOB62wJs%3d> Joel Knighton Cassandra Developer | joel.knigh...@datastax.com<mailto:joel.knigh...@datastax.com> [https://lh3.googleusercontent.com/yF5_lV4h7D2UxEplm05DKDejwQH2rN54kQRIAMwYl3cOFAWVwodcGTBiCsOkOVBSKD644UsZpdwaHZTpN4TH56tTGAjI6X0QAPdSy3BHUHKM_L6-38Z4iZnaju6iTXyRNFFJYYs]<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.linkedin.com%2fcompany%2fdatastax&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7c82fcf2ffa417448f9e3d08d35f06513b%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=TykTq%2f9Ri9iOHG8keMdSRKX%2bu%2bbKR568UrZUzx%2bh1KE%3d>[https://lh6.googleusercontent.com/rfmg_Y2DLoU0blnPfsgAXGhlvS5qFs_Env_XvgglzoC8oYjpMrXkeeGtCOs1n4O-c4hWt7sB6JP_bVoshrvSDRs9d6t2h-94rPgih78BO7eizEHIkojHoFjFlbp9ev6VXowy9Uc]<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.facebook.com%2fdatastax&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7c82fcf2ffa417448f9e3d08d35f06513b%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=etlt12%2b2qtjm6avK18yqqj6Urd2YYY9rRQZ7R9rKL0w%3d>[https://lh5.googleusercontent.com/f51dqYsxdPnH8vFbRcv01-CfYgwLWMRy6h0duHVx20vZdVGofchf9EwXO-QbK2iYu4B_XK39s-CUkTALWWRKAT5h5muJlJDE1G9aP0AS6_CHEehFXHal9QhmRqxEy0APsne2jRY]<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2ftwitter.com%2fdatastax&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7c82fcf2ffa417448f9e3d08d35f06513b%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=M3G4JkUh2k%2f9LN7eTxEFTnN%2f23XhgnbapnGSVLQtnW8%3d>[https://lh4.googleusercontent.com/1nF3jCTQgssSgkR8t8YGl29Xh0m4j6cjnwK-f8MYw3hs0ntHRPekqX7nOgsUTC8pe1skAHyqQJk58mYl1O02CYcT9Dm_QF_bITwZrperb5ufpSCNLVAHRnzWldryaRDe5Q3AsBQ]<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fplus.google.com%2f%2bDatastax%2fabout&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7c82fcf2ffa417448f9e3d08d35f06513b%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=lsc9nihHlgVHFucRp0NwARk7VQ3FBMbQIsvRgUj2580%3d>[https://lh5.googleusercontent.com/vkQxsPuzc0ZSSFU5mxnSHN7WaQN2GKK3hhORrINgHpxl6QB0eReJ3RjSNvKgFTjB85qTrvxHm125957h_vWszsE5GF2sjBJa8_kbEdN8tRZfLbCSZ2JbnrIpNH1r-PHafmFhPEE]<https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2ffeeds.feedburner.com%2fdatastax&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7c82fcf2ffa417448f9e3d08d35f06513b%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=GUnm4ztK3OtETVr7KTZgsOpgA2XGc8skbLHY2Sx0edg%3d>[https://lh4.googleusercontent.com/70iAX8N2qH7_bxPX42x9HswoVfGFFSl4ueEXoYyp4APr3S1O1chNJSVTCVbY0Ta1-IjnqtirbcoFI9SGxkFBncliESx4uz8XHsQ5SsG82WOFDnMMkSKcpe10rr3Lbv9jkIYgQsU]<https://github.com/datastax/>
