Re: Assassinate fails

Hi Anthony, Thanks for your help. I tried to run multiple times in quick succession but it fails with : -- StackTrace -- java.lang.RuntimeException: Endpoint still alive: /192.168.1.18 generation changed while trying to assassinate it at org.apache.cassandra.gms.Gossiper.assassinateEn

OutboundMessagingConnection behavior under high latencies between Cassandra nodes

Hi! I have a design question about the communication layer in Cassandra. Some context first: 1.     I am using Cassandra in a high inter-node communication latency environment, e.g latency between nodes is higher than 60 ms, like in multi-datacenter environments. 2.     Currently, I am

RE: Assassinate fails

Hi Alex, You might have inconsistent data in your system tables. Try setting the consistency level to ALL, then do read query of system tables to force repair. Kenneth Brotman From: Alex [mailto:m...@aca-o.com] Sent: Thursday, April 04, 2019 1:58 AM To: user@cassandra.apache.org Subj

RE: Assassinate fails

This will sound a little silly but, have you tried rolling the cluster? $> nodetool flush; nodetool drain; service cassandra stop $> ps aux | grep ‘cassandra’ # make sure the process actually dies. If not you may need to kill -9 . Check first to see if nodetool can connect first, nodetool gossip

Re: Assassinate fails

How long ago did you remove this host from the cluster? -- Jeff Jirsa > On Apr 4, 2019, at 8:09 AM, Nick Hatfield wrote: > > This will sound a little silly but, have you tried rolling the cluster? > > $> nodetool flush; nodetool drain; service cassandra stop > $> ps aux | grep ‘cassandra’

SStable format change in 3.0.18 ?

Hello ! I have noticed something since I upgraded to cassandra 3.0.18. Before all my Sstable used to be named this way : ``` mc-130817-big-CompressionInfo.db mc-130817-big-Data.db mc-130817-big-Digest.crc32 mc-130817-big-Filter.db mc-130817-big-Index.db mc-130817-big-Statistics.db mc-130817-big-S

Re: SStable format change in 3.0.18 ?

This is CASSANDRA-14861 -- Jeff Jirsa > On Apr 4, 2019, at 8:23 AM, Léo FERLIN SUTTON > wrote: > > Hello ! > > I have noticed something since I upgraded to cassandra 3.0.18. > > Before all my Sstable used to be named this way : > ``` > mc-130817-big-CompressionInfo.db > mc-130817-big-Dat

Re: Assassinate fails

Hi Alex, About previous advices: You might have inconsistent data in your system tables. Try setting the > consistency level to ALL, then do read query of system tables to force > repair. > System tables use the 'LocalStrategy', thus I don't think any repair would happen for the system.* tables

Re: SStable format change in 3.0.18 ?

Hello, I think it was done in the following issue: Sstable min/max metadata can cause data loss (CASSANDRA-14861) https://github.com/apache/cassandra/commit/d60c78358b6f599a83f3c112bfd6ce72c1129c9f src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java

RE: Assassinate fails

The trick below I got from the book Mastering Cassandra. You have to set the consistency to ALL for it to work. I thought you guys knew that one. From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: Thursday, April 04, 2019 8:46 AM To: user cassandra.apache.org Subject: Re: Assassinate fa

Re: SStable format change in 3.0.18 ?

Thank you guys ! On Thu, Apr 4, 2019 at 5:49 PM Dmitry Saprykin wrote: > Hello, > > I think it was done in the following issue: Sstable min/max metadata can > cause data loss (CASSANDRA-14861) > > > https://github.com/apache/cassandra/commit/d60c78358b6f599a83f3c112bfd6ce72c1129c9f > src/java/

Re: Assassinate fails

Ken, Alain is right about the system tables. What you're describing only works on non-local tables. Changing the CL doesn't help with keyspaces that use LocalStrategy. Here's the definition of the system keyspace: CREATE KEYSPACE system WITH replication = {'class': 'LocalStrategy'} AND durable

Re: Assassinate fails

Hi, @ Alain and Kenneth : I use C* for a time series database (KairosDB) ; replication and consistency are set by KairosDB and I would rather not mingle with it. @ Nick and Alain : I have tried to stop / start every node but not with this process. I will try. @ Jeff : I removed (replaced)

RE: Assassinate fails

>From Mastering Cassandra: Forcing read repairs at consistency – ALL The type of repair isn't really part of the Apache Cassandra repair paradigm at all. When it was discovered that a read repair will trigger 100% of the time when a query is run at ALL consistency, this method of repair starte

Re: Assassinate fails

Well, I tried : rolling restart did not work its magic. |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.1.9 26.32 GiB 256 42.8% 76223d4c-9d9f-417f-be27-cebb791cddcc r

Re: Assassinate fails

System != system_auth. On Thu, Apr 4, 2019 at 9:43 AM Kenneth Brotman wrote: > > From Mastering Cassandra: > > > Forcing read repairs at consistency – ALL > > The type of repair isn't really part of the Apache Cassandra repair paradigm > at all. When it was discovered that a read repair will tri

RE: Assassinate fails

Right, could be similar issue, same type of fix though. -Original Message- From: Jon Haddad [mailto:j...@jonhaddad.com] Sent: Thursday, April 04, 2019 9:52 AM To: user@cassandra.apache.org Subject: Re: Assassinate fails System != system_auth. On Thu, Apr 4, 2019 at 9:43 AM Kenneth Brotm

Re: Assassinate fails

No, it can't. As Alain (and I) have said, since the system keyspace is local strategy, it's not replicated, and thus can't be repaired. On Thu, Apr 4, 2019 at 9:54 AM Kenneth Brotman wrote: > > Right, could be similar issue, same type of fix though. > > -Original Message- > From: Jon Had

RE: Assassinate fails

I see; system_auth is a separate keyspace. -Original Message- From: Jon Haddad [mailto:j...@jonhaddad.com] Sent: Thursday, April 04, 2019 10:17 AM To: user@cassandra.apache.org Subject: Re: Assassinate fails No, it can't. As Alain (and I) have said, since the system keyspace is loca

RE: Assassinate fails

Alex, Did you remove the option JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=address_of_dead_node after the node started and then restart the node again? Are you sure there isn't a typo in the file? Ken -Original Message- From: Kenneth Brotman [mailto:kenbrot...@yahoo.com.INVALID]

RE: Assassinate fails

Alex, According to this TLP article http://thelastpickle.com/blog/2018/09/18/assassinate.html : Note that the LEFT status should stick around for 72 hours to ensure all nodes come to the consensus that the node has been removed. So please don’t rush things if that’s the case. Again, it’s only