Removed node is not completely removed

[Tom van den Berge]

I have removed a node with nodetool removenode, which completed ok.
Nodetool status does not list the node anymore.

But since then, Im seeing messages in my other nodes log files referring to
the removed node:

 INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line 968)
InetAddress /10.68.56.200 is now DOWN
 INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java (line
1891) Removing tokens [85070591730234615865843651857942052863] for /
10.68.56.200


These two messages appear every minute.
I've tried nodetool removenode again (Host ID not found) and removenode
force (no token removals in process).
The jmx unsafeAssassinateEndpoint gives a NullPointerException.

What can I do to remove the node entirely?

Change hostname in metrics reporter

[Jayapandian Ponraj]

How to override_hostname for ganglia reporter

ganglia:
  -
period: 300
timeunit: 'SECONDS'
gmondConf: '/etc/ganglia/gmond.conf'
groupPrefix: "cassandra"
predicate:
  color: "white"
  useQualifiedName: true
  patterns:
- "^org.apache.cassandra.metrics.Compaction.+"

unsubscribe

[alessandro oggianu]

unsubscribe

unsubscribe

[Numan Fatih YARCI]

unsubscribe

Re: Removed node is not completely removed

[Carlos Rolo]

Check system.peers table to see if the IP is still there. If so edit the
table and remove the offending IP.

You are probably running into this:
https://issues.apache.org/jira/browse/CASSANDRA-6053

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com

On Wed, Oct 14, 2015 at 12:26 PM, Tom van den Berge <
tom.vandenbe...@gmail.com> wrote:

> I have removed a node with nodetool removenode, which completed ok.
> Nodetool status does not list the node anymore.
>
> But since then, Im seeing messages in my other nodes log files referring
> to the removed node:
>
>  INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line 968)
> InetAddress /10.68.56.200 is now DOWN
>  INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java (line
> 1891) Removing tokens [85070591730234615865843651857942052863] for /
> 10.68.56.200
>
>
> These two messages appear every minute.
> I've tried nodetool removenode again (Host ID not found) and removenode
> force (no token removals in process).
> The jmx unsafeAssassinateEndpoint gives a NullPointerException.
>
> What can I do to remove the node entirely?
>
>
>

-- 


--

Re: unsubscribe

[Amila Paranawithana]

unsubscribe

On Wed, Oct 14, 2015 at 5:57 PM, Numan Fatih YARCI 
wrote:

> unsubscribe
>
>


-- 

*Amila Iroshani Paranawithana , **Senior Software Engineer* *, AdroitLogic
*
| ☎: +94779747398
| ✍: http://amilaparanawithana.blogspot.com
[image: Facebook]  [image:
Twitter]  [image: LinkedIn]
 [image:
Skype] amila.paranawithana
​

Re: Removed node is not completely removed

[Tom van den Berge]

Hi Carlos,

I'm using 2.1.6. The mysterious node is not in the peers table. Any other
ideas?
One of my existing nodes is not present in the system.peers table, though.
Should I be worried?

Regards,
Tom

On Wed, Oct 14, 2015 at 2:27 PM, Carlos Rolo  wrote:

> Check system.peers table to see if the IP is still there. If so edit the
> table and remove the offending IP.
>
> You are probably running into this:
> https://issues.apache.org/jira/browse/CASSANDRA-6053
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
> *
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Wed, Oct 14, 2015 at 12:26 PM, Tom van den Berge <
> tom.vandenbe...@gmail.com> wrote:
>
>> I have removed a node with nodetool removenode, which completed ok.
>> Nodetool status does not list the node anymore.
>>
>> But since then, Im seeing messages in my other nodes log files referring
>> to the removed node:
>>
>>  INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line 968)
>> InetAddress /10.68.56.200 is now DOWN
>>  INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java (line
>> 1891) Removing tokens [85070591730234615865843651857942052863] for /
>> 10.68.56.200
>>
>>
>> These two messages appear every minute.
>> I've tried nodetool removenode again (Host ID not found) and removenode
>> force (no token removals in process).
>> The jmx unsafeAssassinateEndpoint gives a NullPointerException.
>>
>> What can I do to remove the node entirely?
>>
>>
>>
>
> --
>
>
>
>

Re: Removed node is not completely removed

[Laing, Michael]

Remember that the system keyspace uses LocalStrategy: each node has its own
set of system tables. -ml

On Wed, Oct 14, 2015 at 9:17 AM, Tom van den Berge <
tom.vandenbe...@gmail.com> wrote:

> Hi Carlos,
>
> I'm using 2.1.6. The mysterious node is not in the peers table. Any other
> ideas?
> One of my existing nodes is not present in the system.peers table, though.
> Should I be worried?
>
> Regards,
> Tom
>
> On Wed, Oct 14, 2015 at 2:27 PM, Carlos Rolo  wrote:
>
>> Check system.peers table to see if the IP is still there. If so edit the
>> table and remove the offending IP.
>>
>> You are probably running into this:
>> https://issues.apache.org/jira/browse/CASSANDRA-6053
>>
>> Regards,
>>
>> Carlos Juzarte Rolo
>> Cassandra Consultant
>>
>> Pythian - Love your data
>>
>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>> *
>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>> www.pythian.com
>>
>> On Wed, Oct 14, 2015 at 12:26 PM, Tom van den Berge <
>> tom.vandenbe...@gmail.com> wrote:
>>
>>> I have removed a node with nodetool removenode, which completed ok.
>>> Nodetool status does not list the node anymore.
>>>
>>> But since then, Im seeing messages in my other nodes log files referring
>>> to the removed node:
>>>
>>>  INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line 968)
>>> InetAddress /10.68.56.200 is now DOWN
>>>  INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java (line
>>> 1891) Removing tokens [85070591730234615865843651857942052863] for /
>>> 10.68.56.200
>>>
>>>
>>> These two messages appear every minute.
>>> I've tried nodetool removenode again (Host ID not found) and removenode
>>> force (no token removals in process).
>>> The jmx unsafeAssassinateEndpoint gives a NullPointerException.
>>>
>>> What can I do to remove the node entirely?
>>>
>>>
>>>
>>
>> --
>>
>>
>>
>>
>

Re: Removed node is not completely removed

[Tom van den Berge]

Thanks for that Michael, I did not know that. However, the node is not
listed in the system.peers table on any node, so it seems that the problem
is not in this table.



On Wed, Oct 14, 2015 at 3:30 PM, Laing, Michael 
wrote:

> Remember that the system keyspace uses LocalStrategy: each node has its
> own set of system tables. -ml
>
> On Wed, Oct 14, 2015 at 9:17 AM, Tom van den Berge <
> tom.vandenbe...@gmail.com> wrote:
>
>> Hi Carlos,
>>
>> I'm using 2.1.6. The mysterious node is not in the peers table. Any other
>> ideas?
>> One of my existing nodes is not present in the system.peers table,
>> though. Should I be worried?
>>
>> Regards,
>> Tom
>>
>> On Wed, Oct 14, 2015 at 2:27 PM, Carlos Rolo  wrote:
>>
>>> Check system.peers table to see if the IP is still there. If so edit the
>>> table and remove the offending IP.
>>>
>>> You are probably running into this:
>>> https://issues.apache.org/jira/browse/CASSANDRA-6053
>>>
>>> Regards,
>>>
>>> Carlos Juzarte Rolo
>>> Cassandra Consultant
>>>
>>> Pythian - Love your data
>>>
>>> rolo@pythian | Twitter: cjrolo | Linkedin: 
>>> *linkedin.com/in/carlosjuzarterolo
>>> *
>>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>>> www.pythian.com
>>>
>>> On Wed, Oct 14, 2015 at 12:26 PM, Tom van den Berge <
>>> tom.vandenbe...@gmail.com> wrote:
>>>
 I have removed a node with nodetool removenode, which completed ok.
 Nodetool status does not list the node anymore.

 But since then, Im seeing messages in my other nodes log files
 referring to the removed node:

  INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line 968)
 InetAddress /10.68.56.200 is now DOWN
  INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java
 (line 1891) Removing tokens [85070591730234615865843651857942052863] for /
 10.68.56.200


 These two messages appear every minute.
 I've tried nodetool removenode again (Host ID not found) and removenode
 force (no token removals in process).
 The jmx unsafeAssassinateEndpoint gives a NullPointerException.

 What can I do to remove the node entirely?



>>>
>>> --
>>>
>>>
>>>
>>>
>>
>

Re: Removed node is not completely removed

[Sebastian Estevez]

We still keep endpoints in memory. Not sure how you git to this state but
try a rolling restart.
On Oct 14, 2015 9:43 AM, "Tom van den Berge" 
wrote:

> Thanks for that Michael, I did not know that. However, the node is not
> listed in the system.peers table on any node, so it seems that the problem
> is not in this table.
>
>
>
> On Wed, Oct 14, 2015 at 3:30 PM, Laing, Michael  > wrote:
>
>> Remember that the system keyspace uses LocalStrategy: each node has its
>> own set of system tables. -ml
>>
>> On Wed, Oct 14, 2015 at 9:17 AM, Tom van den Berge <
>> tom.vandenbe...@gmail.com> wrote:
>>
>>> Hi Carlos,
>>>
>>> I'm using 2.1.6. The mysterious node is not in the peers table. Any
>>> other ideas?
>>> One of my existing nodes is not present in the system.peers table,
>>> though. Should I be worried?
>>>
>>> Regards,
>>> Tom
>>>
>>> On Wed, Oct 14, 2015 at 2:27 PM, Carlos Rolo  wrote:
>>>
 Check system.peers table to see if the IP is still there. If so edit
 the table and remove the offending IP.

 You are probably running into this:
 https://issues.apache.org/jira/browse/CASSANDRA-6053

 Regards,

 Carlos Juzarte Rolo
 Cassandra Consultant

 Pythian - Love your data

 rolo@pythian | Twitter: cjrolo | Linkedin: 
 *linkedin.com/in/carlosjuzarterolo
 *
 Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
 www.pythian.com

 On Wed, Oct 14, 2015 at 12:26 PM, Tom van den Berge <
 tom.vandenbe...@gmail.com> wrote:

> I have removed a node with nodetool removenode, which completed ok.
> Nodetool status does not list the node anymore.
>
> But since then, Im seeing messages in my other nodes log files
> referring to the removed node:
>
>  INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line
> 968) InetAddress /10.68.56.200 is now DOWN
>  INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java
> (line 1891) Removing tokens [85070591730234615865843651857942052863] for /
> 10.68.56.200
>
>
> These two messages appear every minute.
> I've tried nodetool removenode again (Host ID not found) and
> removenode force (no token removals in process).
> The jmx unsafeAssassinateEndpoint gives a NullPointerException.
>
> What can I do to remove the node entirely?
>
>
>

 --




>>>
>>
>

unsubscribe

[Rampally, Parthasaradhi - Fairfax, VA - Contractor]

unsubscribe

Partha Rampally

Re: unsubscribe

[Alain RODRIGUEZ]

>From http://cassandra.apache.org/#lists --> just email
user-unsubscr...@cassandra.apache.org


Alain

2015-10-14 14:29 GMT+02:00 Amila Paranawithana :

> unsubscribe
>
> On Wed, Oct 14, 2015 at 5:57 PM, Numan Fatih YARCI 
> wrote:
>
>> unsubscribe
>>
>>
>
>
> --
>
> *Amila Iroshani Paranawithana , **Senior Software Engineer* *,
> AdroitLogic *
> | ☎: +94779747398
> | ✍: http://amilaparanawithana.blogspot.com
> [image: Facebook]  [image:
> Twitter]  [image: LinkedIn]
>  [image:
> Skype] amila.paranawithana
> ​
>

Re: unsubscribe

[rock zhang]

unsubscribe
> On Oct 14, 2015, at 7:28 AM, Alain RODRIGUEZ  wrote:
> 
> From http://cassandra.apache.org/#lists  
> --> just email user-unsubscr...@cassandra.apache.org 
> 
> 
> 
> Alain
> 
> 2015-10-14 14:29 GMT+02:00 Amila Paranawithana  >:
> unsubscribe
> 
> On Wed, Oct 14, 2015 at 5:57 PM, Numan Fatih YARCI  > wrote:
> unsubscribe
> 
> 
> 
> 
> -- 
> 
> Amila Iroshani Paranawithana , Senior Software Engineer , AdroitLogic 
>  
> | ☎: +94779747398  
> | ✍: http://amilaparanawithana.blogspot.com 
> 
>    
>   
>   
> amila.paranawithana
> ​
> 

Re: unsubscribe

[Gene]

unsubscribe
afmelden
abbestellen
ignorer cette


On Wed, Oct 14, 2015 at 8:24 AM, rock zhang  wrote:

> unsubscribe
>
> On Oct 14, 2015, at 7:28 AM, Alain RODRIGUEZ  wrote:
>
> From http://cassandra.apache.org/#lists --> just email
> user-unsubscr...@cassandra.apache.org
>
>
> Alain
>
> 2015-10-14 14:29 GMT+02:00 Amila Paranawithana :
>
>> unsubscribe
>>
>> On Wed, Oct 14, 2015 at 5:57 PM, Numan Fatih YARCI > > wrote:
>>
>>> unsubscribe
>>>
>>>
>>
>>
>> --
>>
>> *Amila Iroshani Paranawithana , **Senior Software Engineer* *,
>> AdroitLogic *
>> | ☎: +94779747398
>> | ✍: http://amilaparanawithana.blogspot.com
>> [image: Facebook]  [image:
>> Twitter]  [image: LinkedIn]
>>  [image:
>> Skype] amila.paranawithana
>> ​
>>
>
>
>

Collections (MAP) data in Column Family

[Saladi Naidu]

We are running Apache Cassandra 2.1.9. In one of our Column Family, we have MAP 
type column. We are seeing unusual data size of the column family (SSTables) 
with few 1000's of rows, while debugging, I looked at one of the SSTable and I 
see some unusual data in it
Below is JSON of one Row Key data
1. There is usual Column name, Key-Value pair and TS for the MAP - all_products 
column name 2. After Key Value pair,  I see Cluster Column style data in MAP 
with a "t" marker in between, this is literally repeated millions of cells - 
all_products:_","all_products:!",1442797965371999,"t",1442797965
Any clues on what is happening here? I know "d" marker for marked for delete, 
"e" marker for TTL but dont know what "t" marker is for? 
[{"key": "55736100", "cells": [["","",1444101633184000],           
["active","false",1444101633184000],           
["all_products:_","all_products:!",1442797965371999,"t",1442797965],           
["all_products:_","all_products:!",1442806687091999,"t",1442806687],           
["all_products:_","all_products:!",1443410022982999,"t",1443410022],           
["all_products:_","all_products:!",1443410595224999,"t",1443410595],           
["all_products:_","all_products:!",1443679978903999,"t",1443679978],           
["all_products:_","all_products:!",1444011801906999,"t",1444011801],           
["all_products:_","all_products:!",1444101633183999,"t",1444101633],           
["all_products:3135393730323130305f63735f435a","313539373032313030",1444101633184000],
           
["all_products:3135393730323130305f64655f4154","313539373032313030",1444101633184000],
           
["all_products:3135393730323130305f64655f4348","313539373032313030",1444101633184000],
           
["all_products:3135393730323130305f64655f4445","313539373032313030",1444101633184000],
           
["all_products:3135393730323130305f656e5f4348","313539373032313030",1444101633184000],
.["all_products:3233393238333430305f69745f4348","323339323833343030",1444101633184000],
           ["all_products:_","all_products:!",1442797965371999,"t",1442797965], 
          ["all_products:_","all_products:!",1442797965371999,"t",1442797965],  
         ["all_products:_","all_products:!",1442806687091999,"t",1442806687],   
        ["all_products:_","all_products:!",1442797965371999,"t",1442797965],    
       ["all_products:_","all_products:!",1442797965371999,"t",1442797965],     
      ["all_products:_","all_products:!",1442806687091999,"t",1442806687],      
     ["all_products:_","all_products:!",1442797965371999,"t",1442797965],       
    ["all_products:_","all_products:!",1442797965371999,"t",1442797965],        
   ["all_products:_","all_products:!",1442806687091999,"t",1442806687],         
  ["all_products:_","all_products:!",1442797965371999,"t",1442797965],          
 ["all_products:_","all_products:!",1442797965371999,"t",1442797965],           
["all_products:_","all_products:!",1442806687091999,"t",1442806687],           
["all_products:_","all_products:!",1442797965371999,"t",1442797965],           
["all_products:_","all_products:!",1442797965371999,"t",1442797965],           
["all_products:_","all_products:!",1442806687091999,"t",1442806687],           
["all_products:_","all_products:!",1442797965371999,"t",1442797965],           
["all_products:_","all_products:!",1442797965371999,"t",1442797965],           
["all_products:_","all_products:!",1442806687091999,"t",1442806687],           
["all_products:_","all_products:!",1442797965371999,"t",1442797965],           
["all_products:_","all_products:!",1442797965371999,"t",1442797965],           
["all_products:_","all_products:!",1442806687091999,"t",1442806687],           
["all_products:_","all_products:!",1442797965371999,"t",1442797965],           
["all_products:_","all_products:!",1442797965371999,"t",1442797965],           
["all_products:_","all_products:!",1442806687091999,"t",1442806687],

 Naidu Saladi 

Re: Collections (MAP) data in Column Family

[Robert Coli]

On Wed, Oct 14, 2015 at 11:24 AM, Saladi Naidu 
wrote:

> We are running Apache Cassandra 2.1.9. In one of our Column Family, we
> have MAP type column. We are seeing unusual data size of the column family
> (SSTables) with few 1000's of rows, while debugging, I looked at one of the
> SSTable and I see some unusual data in it
>
...

> 2. After Key Value pair,  I see Cluster Column style data in MAP with a
> "t" marker in between, this is literally repeated millions of cells -
> *all_products:_","all_products:!",1442797965371999,"t",1442797965*
>

(thx to driftx@IRC for the link to...)

https://issues.apache.org/jira/browse/CASSANDRA-10505 /
https://issues.apache.org/jira/browse/CASSANDRA-7953

" Range tombstones not purged during compaction / Duplicate range
tombstones "

It looks like you will need to wait for the next 2.1.x release for a fix. :)

=Rob

Re: reiserfs - DirectoryNotEmptyException

[Michael Shuler]

On 10/13/2015 01:58 PM, Modha, Digant wrote:

I am running Cassandra 2.1.10 and noticed intermittent
DirectoryNotEmptyExceptions during repair.  My cassandra data drive
is reiserfs.


Why? I'm genuinely interested in this filesystem selection, since it is
unmaintained, has been dropped from some mainstream linux distributions,
and some may call it "dead". ;)


I noticed that on reiserfs wiki site
https://en.m.wikipedia.org/wiki/ReiserFS#Criticism, it states that
unlink operation is not synchronous. Is that the reason for the
exception below:

ERROR [ValidationExecutor:137] 2015-10-13 00:46:30,759
CassandraDaemon.java:227 - Exception in thread
Thread[ValidationExecutor:137,1,main]

org.apache.cassandra.io.FSWriteError:
java.nio.file.DirectoryNotEmptyException:

at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135)

~[apache-cassandra-2.1.10.jar:2.1.10]

<...>

This seems like a reasonable explanation. Using a modern filesystem like
ext4 or xfs would certainly be helpful in getting you within the realm
of a "common" hardware setup.

https://wiki.apache.org/cassandra/CassandraHardware
https://www.safaribooksonline.com/library/view/cassandra-high-performance/9781849515122/ch04s06.html

I think Al Tobey had a slide deck on filesystem tuning for C*, but I 
didn't find it quickly.


--
Kind regards,
Michael

Re: reiserfs - DirectoryNotEmptyException

[Michael Shuler]

On 10/14/2015 05:02 PM, Michael Shuler wrote:

I think Al Tobey had a slide deck on filesystem tuning for C*, but I
didn't find it quickly.


Right! Al included this in his tuning guide:

https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html

--
Michael

Re: Removed node is not completely removed

[Tom van den Berge]

Thanks Sebastian, a restart solved the problem!


On Wed, Oct 14, 2015 at 3:46 PM, Sebastian Estevez <
sebastian.este...@datastax.com> wrote:

> We still keep endpoints in memory. Not sure how you git to this state but
> try a rolling restart.
> On Oct 14, 2015 9:43 AM, "Tom van den Berge" 
> wrote:
>
>> Thanks for that Michael, I did not know that. However, the node is not
>> listed in the system.peers table on any node, so it seems that the problem
>> is not in this table.
>>
>>
>>
>> On Wed, Oct 14, 2015 at 3:30 PM, Laing, Michael <
>> michael.la...@nytimes.com> wrote:
>>
>>> Remember that the system keyspace uses LocalStrategy: each node has its
>>> own set of system tables. -ml
>>>
>>> On Wed, Oct 14, 2015 at 9:17 AM, Tom van den Berge <
>>> tom.vandenbe...@gmail.com> wrote:
>>>
 Hi Carlos,

 I'm using 2.1.6. The mysterious node is not in the peers table. Any
 other ideas?
 One of my existing nodes is not present in the system.peers table,
 though. Should I be worried?

 Regards,
 Tom

 On Wed, Oct 14, 2015 at 2:27 PM, Carlos Rolo  wrote:

> Check system.peers table to see if the IP is still there. If so edit
> the table and remove the offending IP.
>
> You are probably running into this:
> https://issues.apache.org/jira/browse/CASSANDRA-6053
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: 
> *linkedin.com/in/carlosjuzarterolo
> *
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Wed, Oct 14, 2015 at 12:26 PM, Tom van den Berge <
> tom.vandenbe...@gmail.com> wrote:
>
>> I have removed a node with nodetool removenode, which completed ok.
>> Nodetool status does not list the node anymore.
>>
>> But since then, Im seeing messages in my other nodes log files
>> referring to the removed node:
>>
>>  INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line
>> 968) InetAddress /10.68.56.200 is now DOWN
>>  INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java
>> (line 1891) Removing tokens [85070591730234615865843651857942052863] for 
>> /
>> 10.68.56.200
>>
>>
>> These two messages appear every minute.
>> I've tried nodetool removenode again (Host ID not found) and
>> removenode force (no token removals in process).
>> The jmx unsafeAssassinateEndpoint gives a NullPointerException.
>>
>> What can I do to remove the node entirely?
>>
>>
>>
>
> --
>
>
>
>

>>>
>>