Update cassandra from perl 5.8

[Jacob Rhoden]

Hi,

We’ve got a legacy application on Perl 5.8, don’t want to upgrade the server 
for fear of something breaking. However for redundancy, It would be great if I 
could update the app so it writes to both the existing database, and cassandra. 
The Perl drivers appear to require perl 5.10+

https://github.com/mkjellman/perlcassa
https://metacpan.org/pod/Net::Async::CassandraCQL

Just thought I would ask the list in the off chance that someone might know of 
an alternate way to solve this problem or shoehorn a perl driver onto this 
server.

One option is nightly bulk data export, but I’d rather avoid that if possible.

Thanks,
Jacob

Fwd: Unable to gossip with any seeds

[Sundeep Kambhampati]

Hi,

I am trying to setup multi-node Cassandra cluster (2 nodes). I am
using apache-cassandra-2.0.4. I am able to start Cassandra on the seed
node. But, when I am trying to start it on the other node it starts and
fails in few seconds. I can see the following in my error log:

"ERROR 03:23:56,915 Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds "


I am able to telnet from node 1 to node 0.

telnet 10.2.252.0 9000
Trying 10.2.252.0...
Connected to 10.2.252.0.
Escape character is '^]'.
^]
Connection closed by foreign host.


*cassandra.yaml*

node 0: (sk.r252.0)(seed)

cluster_name: 'DataCluster'
num_tokens: 256
initial_token: 0
seeds: "sk.r252.0"
storage_port: 9000
ssl_storage_port: 9001
listen_address: sk.r252.0
rpc_address: 0.0.0.0
rpc_port: 8192
endpoint_snitch: RackInferringSnitch

node 1: (sk.r252.1)
cluster_name: 'DataCluster'
num_tokens: 256
initial_token: 4611686018427387904
seeds: "sk.r252.0"
storage_port: 9000
ssl_storage_port: 9001
listen_address: sk.r252.1
rpc_address: 0.0.0.0
rpc_port: 8192
endpoint_snitch: RackInferringSnitch


When I am trying to start Cassandra on node 1 it fails and the log shows:

 INFO 03:23:25,284 Loading persisted ring state
 INFO 03:23:25,564 Starting Messaging Service on port 9000
 INFO 03:23:25,797 Handshaking version with sk.r252.0/10.2.252.0
ERROR 03:23:56,915 Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at
org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
java.lang.RuntimeException: Unable to gossip with any seeds
at
org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
Exception encountered during startup: Unable to gossip with any seeds
ERROR 03:24:02,213 Exception in thread
Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:349)
at
org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:364)
at
org.apache.cassandra.service.StorageService.access$000(StorageService.java:97)
at
org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Unknown Source)

*bin/nodetool status*


Datacenter: 0
=
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address   Load   Owns (effective)  Host ID
  TokenRack
UN  10.0.2.1  35.8 KB100.0%
 0ae766db-fcb5-481b-8120-550b672fa9e7  0
 2


Can some please help me fixing this error?


Thank you,
Sundeep

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

[olek.stas...@gmail.com]

Hi All,
We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via
1.2.10). Probably after upgradesstable  (but it's only a guess,
because we noticed problem few weeks later), some rows became
tombstoned. They just disappear from results of queries. After
inverstigation I've noticed, that they are reachable via sstable2json.
Example output for "non-existent" row:

{"key": "6e6e37716c6d665f6f61695f6463","metadata": {"deletionInfo":
{"markedForDeleteAt":2201170739199,"localDeletionTime":0}},"columns":
[["DATA","3c6f61695f64633a64(...)",1357677928108]]}
]

If I understand correctly row is marked as deleted with timestamp in
the far future, but it's still on the disk. Also localDeletionTime is
set to 0, which may means, that it's kind of internal bug, not effect
of client error. So my question is: is it true, that upgradesstable
may do soemthing like that? How to find reasons for such strange
cassandra behaviour? Is there any option of recovering such strange
marked nodes?
This problem touches about 500K rows of all 14M in our database, so
the percentage is quite big.
best regards
Aleksander

2013-12-12 Robert Coli :
> On Wed, Dec 11, 2013 at 6:27 AM, Mathijs Vogelzang 
> wrote:
>>
>> When I use sstable2json on the sstable on the destination cluster, it has
>> "metadata": {"deletionInfo":
>> {"markedForDeleteAt":1796952039620607,"localDeletionTime":0}}, whereas
>> it doesn't have that in the source sstable.
>> (Yes, this is a timestamp far into the future. All our hosts are
>> properly synced through ntp).
>
>
> This seems like a bug in sstableloader, I would report it on JIRA.
>
>>
>> Naturally, copying the data again doesn't work to fix it, as the
>> tombstone is far in the future. Apart from not having this happen at
>> all, how can it be fixed?
>
>
> Briefly, you'll want to purge that tombstone and then reload the data with a
> reasonable timestamp.
>
> Dealing with rows with data (and tombstones) in the far future is described
> in detail here :
>
> http://thelastpickle.com/blog/2011/12/15/Anatomy-of-a-Cassandra-Partition.html
>
> =Rob
>

Re: Unable to gossip with any seeds

[Chiranjeevi Ravilla]

Hi Sundeep,

Can you please confirm, are you configuring two nodes in different Datacenters?

If you are configuring on single datacenter with two nodes,then please change 
the endpoint_snitch  from  RackInferringSnitch to SimpleSnitch and restart the 
clusters.

Regards,
Chiru

On 03-Feb-2014, at 2:17 PM, Sundeep Kambhampati  wrote:

> Hi,
> 
> I am trying to setup multi-node Cassandra cluster (2 nodes). I am using 
> apache-cassandra-2.0.4. I am able to start Cassandra on the seed node. But, 
> when I am trying to start it on the other node it starts and fails in few 
> seconds. I can see the following in my error log: 
> 
> "ERROR 03:23:56,915 Exception encountered during startup
> java.lang.RuntimeException: Unable to gossip with any seeds "
> 
> 
> I am able to telnet from node 1 to node 0.
> 
> telnet 10.2.252.0 9000
> Trying 10.2.252.0...
> Connected to 10.2.252.0.
> Escape character is '^]'.
> ^]
> Connection closed by foreign host.
>  
> 
> cassandra.yaml
> 
> node 0: (sk.r252.0)(seed) 
> 
> cluster_name: 'DataCluster'
> num_tokens: 256
> initial_token: 0
> seeds: "sk.r252.0"
> storage_port: 9000
> ssl_storage_port: 9001
> listen_address: sk.r252.0
> rpc_address: 0.0.0.0
> rpc_port: 8192
> endpoint_snitch: RackInferringSnitch
> 
> node 1: (sk.r252.1) 
> cluster_name: 'DataCluster'
> num_tokens: 256
> initial_token: 4611686018427387904
> seeds: "sk.r252.0"
> storage_port: 9000
> ssl_storage_port: 9001
> listen_address: sk.r252.1
> rpc_address: 0.0.0.0
> rpc_port: 8192
> endpoint_snitch: RackInferringSnitch
> 
> 
> When I am trying to start Cassandra on node 1 it fails and the log shows:
> 
>  INFO 03:23:25,284 Loading persisted ring state
>  INFO 03:23:25,564 Starting Messaging Service on port 9000
>  INFO 03:23:25,797 Handshaking version with sk.r252.0/10.2.252.0
> ERROR 03:23:56,915 Exception encountered during startup
> java.lang.RuntimeException: Unable to gossip with any seeds
> at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
> at 
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
> at 
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
> java.lang.RuntimeException: Unable to gossip with any seeds
> at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
> at 
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
> at 
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
> Exception encountered during startup: Unable to gossip with any seeds
> ERROR 03:24:02,213 Exception in thread 
> Thread[StorageServiceShutdownHook,5,main]
> java.lang.NullPointerException
> at 
> org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:349)
> at 
> org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:364)
> at 
> org.apache.cassandra.service.StorageService.access$000(StorageService.java:97)
> at 
> org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
> at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.lang.Thread.run(Unknown Source)
> 
> bin/nodetool status
> 
> 
> Datacenter: 0
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address   Load   Owns (effective)  Host ID
>TokenRack
> UN  10.0.2.1  35.8 KB100.0%
> 0ae766db-fcb5-481b-8120-550b672fa9e7  0   
>  2
> 
> 
> Can some please help me fixing this error? 
> 
> 
> Thank you,
> Sundeep
> 
> 

Fwd: Unable to gossip with any seeds

[Desimpel, Ignace]

Same problem here. But in my case the same configuration does start up when I 
just retry it a second time.

Some more info on my specific configuration :

Environment : linux, cassandra 2.0.4, 3 node, embedded, byte ordered, LCS

When I add a node to the existing 3 node cluster I sometimes get the exception 
'Unable to gossip with any seeds ' listed below. If I just restart it without 
any change then mostly it works. Must be some timing issue.

The Cassandra at that time is configured using the Cassandra.yaml file
with the auto_bootstrap set true
and the initial_token set to something like : 00f35256, 041e692a, 0562d8b2, 
0930274a, 0b16ce96, 0c5b3e1e, 10cac47a, 12b16bc6, 13f5db4e, 186561aa, 1907996e, 
1c32b042, 1e19578e ..

The two seeds configured in this yaml are 10.164.8.250 and 10.164.8.249 and 
these are up and running.
The new node to add has ip 10.164.8.93

At the time of the exception, I do not get the gossip message 'Handshaking 
version with /10.164.8.93' on the seeds.
If the exception does not occurs, then I do get that gossip message 
'Handshaking version with /10.164.8.93' on the seed

2014-01-31 13:40:36.380 Loading persisted ring state
2014-01-31 13:40:36.386 Starting Messaging Service on port 9804
2014-01-31 13:40:36.408 Handshaking version with /10.164.8.250
2014-01-31 13:40:36.408 Handshaking version with /10.164.8.249
2014-01-31 13:41:07.415 Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at 
org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160) 
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
at 
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
 ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
at 
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
 ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
at 
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) 
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
at 
org.apache.cassandra.service.StorageService.initServer(StorageService.java:485) 
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346) 
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461) 
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
at 
be.landc.services.search.server.db.baseserver.indexsearch.store.cassandra.CassandraStore$CassThread.startUpCassandra(CassandraStore.java:469)
 [landc-services-search-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT-87937]
at 
be.landc.services.search.server.db.baseserver.indexsearch.store.cassandra.CassandraStore$CassThread.run(CassandraStore.java:460)
 [landc-services-search-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT-87937]
java.lang.RuntimeException: Unable to gossip with any seeds
at 
org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
at 
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
at 
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
at 
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
at 
org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
at 
be.landc.services.search.server.db.baseserver.indexsearch.store.cassandra.CassandraStore$CassThread.startUpCassandra(CassandraStore.java:469)
at 
be.landc.services.search.server.db.baseserver.indexsearch.store.cassandra.CassandraStore$CassThread.run(CassandraStore.java:460)
Exception encountered during startup: Unable to gossip with any seeds
2014-01-31 13:41:07.419 Exception in thread 
Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException: null
at 
org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:349)
 ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
at 
org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:364)
 ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
at 
org.apache.cassandra.service.StorageService.access$3(StorageService.java:361) 
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
at 
org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
 ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedR

Re: Unable to gossip with any seeds

[Sundeep Kambhampati]

Thank you Chiru for the reply. I am configuring single datacenter. I
changed it to SimpleSnitch. However, I am getting the same error.

-Sundeep


On Mon, Feb 3, 2014 at 3:58 AM, Chiranjeevi Ravilla  wrote:

> Hi Sundeep,
>
> Can you please confirm, are you configuring two nodes in different
> Datacenters?
>
> If you are configuring on single datacenter with two nodes,then please
> change the endpoint_snitch  from  RackInferringSnitch to SimpleSnitch and
> restart the clusters.
>
> Regards,
> Chiru
>
> On 03-Feb-2014, at 2:17 PM, Sundeep Kambhampati 
> wrote:
>
> Hi,
>
> I am trying to setup multi-node Cassandra cluster (2 nodes). I am
> using apache-cassandra-2.0.4. I am able to start Cassandra on the seed
> node. But, when I am trying to start it on the other node it starts and
> fails in few seconds. I can see the following in my error log:
>
> "ERROR 03:23:56,915 Exception encountered during startup
> java.lang.RuntimeException: Unable to gossip with any seeds "
>
>
> I am able to telnet from node 1 to node 0.
>
> telnet 10.2.252.0 9000
> Trying 10.2.252.0...
> Connected to 10.2.252.0.
> Escape character is '^]'.
> ^]
> Connection closed by foreign host.
>
>
> *cassandra.yaml*
>
> node 0: (sk.r252.0)(seed)
>
> cluster_name: 'DataCluster'
> num_tokens: 256
> initial_token: 0
> seeds: "sk.r252.0"
> storage_port: 9000
> ssl_storage_port: 9001
> listen_address: sk.r252.0
> rpc_address: 0.0.0.0
> rpc_port: 8192
> endpoint_snitch: RackInferringSnitch
>
> node 1: (sk.r252.1)
> cluster_name: 'DataCluster'
> num_tokens: 256
> initial_token: 4611686018427387904
> seeds: "sk.r252.0"
> storage_port: 9000
> ssl_storage_port: 9001
> listen_address: sk.r252.1
> rpc_address: 0.0.0.0
> rpc_port: 8192
> endpoint_snitch: RackInferringSnitch
>
>
> When I am trying to start Cassandra on node 1 it fails and the log shows:
>
>  INFO 03:23:25,284 Loading persisted ring state
>  INFO 03:23:25,564 Starting Messaging Service on port 9000
>  INFO 03:23:25,797 Handshaking version with sk.r252.0/10.2.252.0
> ERROR 03:23:56,915 Exception encountered during startup
> java.lang.RuntimeException: Unable to gossip with any seeds
> at
> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
> at
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
>  at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
> java.lang.RuntimeException: Unable to gossip with any seeds
> at
> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
> at
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
> Exception encountered during startup: Unable to gossip with any seeds
> ERROR 03:24:02,213 Exception in thread
> Thread[StorageServiceShutdownHook,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:349)
> at
> org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:364)
> at
> org.apache.cassandra.service.StorageService.access$000(StorageService.java:97)
> at
> org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.lang.Thread.run(Unknown Source)
>
> *bin/nodetool status*
>
>
> Datacenter: 0
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address   Load   Owns (effective)  Host ID
>   TokenRack
> UN  10.0.2.1  35.8 KB100.0%
>  0ae766db-fcb5-481b-8120-550b672fa9e7  0
>  2
>
>
> Can some please help me fixing this error?
>
>
> Thank you,
> Sundeep
>
>
>
>

Re: Fwd: Unable to gossip with any seeds

[Ilya Sviridov]

Hello Sundeep

It seems that in both configs of your nodes you are using the same hosname
as seeds value.

You have to enumerate all nodes in your cluster.

Best wishes,
Ilya
On Feb 3, 2014 10:47 AM, "Sundeep Kambhampati" 
wrote:

> Hi,
>
> I am trying to setup multi-node Cassandra cluster (2 nodes). I am
> using apache-cassandra-2.0.4. I am able to start Cassandra on the seed
> node. But, when I am trying to start it on the other node it starts and
> fails in few seconds. I can see the following in my error log:
>
> "ERROR 03:23:56,915 Exception encountered during startup
> java.lang.RuntimeException: Unable to gossip with any seeds "
>
>
> I am able to telnet from node 1 to node 0.
>
> telnet 10.2.252.0 9000
> Trying 10.2.252.0...
> Connected to 10.2.252.0.
> Escape character is '^]'.
> ^]
> Connection closed by foreign host.
>
>
> *cassandra.yaml*
>
> node 0: (sk.r252.0)(seed)
>
> cluster_name: 'DataCluster'
> num_tokens: 256
> initial_token: 0
> seeds: "sk.r252.0"
> storage_port: 9000
> ssl_storage_port: 9001
> listen_address: sk.r252.0
> rpc_address: 0.0.0.0
> rpc_port: 8192
> endpoint_snitch: RackInferringSnitch
>
> node 1: (sk.r252.1)
> cluster_name: 'DataCluster'
> num_tokens: 256
> initial_token: 4611686018427387904
> seeds: "sk.r252.0"
> storage_port: 9000
> ssl_storage_port: 9001
> listen_address: sk.r252.1
> rpc_address: 0.0.0.0
> rpc_port: 8192
> endpoint_snitch: RackInferringSnitch
>
>
> When I am trying to start Cassandra on node 1 it fails and the log shows:
>
>  INFO 03:23:25,284 Loading persisted ring state
>  INFO 03:23:25,564 Starting Messaging Service on port 9000
>  INFO 03:23:25,797 Handshaking version with sk.r252.0/10.2.252.0
> ERROR 03:23:56,915 Exception encountered during startup
> java.lang.RuntimeException: Unable to gossip with any seeds
> at
> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
> at
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
>  at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
> java.lang.RuntimeException: Unable to gossip with any seeds
> at
> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
> at
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
> Exception encountered during startup: Unable to gossip with any seeds
> ERROR 03:24:02,213 Exception in thread
> Thread[StorageServiceShutdownHook,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:349)
> at
> org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:364)
> at
> org.apache.cassandra.service.StorageService.access$000(StorageService.java:97)
> at
> org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.lang.Thread.run(Unknown Source)
>
> *bin/nodetool status*
>
>
> Datacenter: 0
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address   Load   Owns (effective)  Host ID
>   TokenRack
> UN  10.0.2.1  35.8 KB100.0%
>  0ae766db-fcb5-481b-8120-550b672fa9e7  0
>  2
>
>
> Can some please help me fixing this error?
>
>
> Thank you,
> Sundeep
>
>
>

Re: Fwd: Unable to gossip with any seeds

[Duncan Sands]

Hi Ilya,

On 03/02/14 10:49, Ilya Sviridov wrote:

Hello Sundeep

It seems that in both configs of your nodes you are using the same hosname as
seeds value.

You have to enumerate all nodes in your cluster.


not so!  If all nodes N1, N2, ... use the same node N0 as a seed, then by 
gossiping with N0 they discover the existence of all the other nodes, because N0 
tells them about them.


The problem with only using one seed node is: what if N0 is down when you start 
a new node?


Ciao, Duncan.



Best wishes,
Ilya

On Feb 3, 2014 10:47 AM, "Sundeep Kambhampati" mailto:satyasunde...@gmail.com>> wrote:

Hi,

I am trying to setup multi-node Cassandra cluster (2 nodes). I am
using apache-cassandra-2.0.4. I am able to start Cassandra on the seed node.
But, when I am trying to start it on the other node it starts and fails in
few seconds. I can see the following in my error log:

"ERROR 03:23:56,915 Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds "


I am able to telnet from node 1 to node 0.

telnet 10.2.252.0 9000
Trying 10.2.252.0...
Connected to 10.2.252.0.
Escape character is '^]'.
^]
Connection closed by foreign host.


*cassandra.yaml*

node 0: (sk.r252.0)(seed)

cluster_name: 'DataCluster'
num_tokens: 256
initial_token: 0
seeds: "sk.r252.0"
storage_port: 9000
ssl_storage_port: 9001
listen_address: sk.r252.0
rpc_address: 0.0.0.0
rpc_port: 8192
endpoint_snitch: RackInferringSnitch

node 1: (sk.r252.1)
cluster_name: 'DataCluster'
num_tokens: 256
initial_token: 4611686018427387904
seeds: "sk.r252.0"
storage_port: 9000
ssl_storage_port: 9001
listen_address: sk.r252.1
rpc_address: 0.0.0.0
rpc_port: 8192
endpoint_snitch: RackInferringSnitch


When I am trying to start Cassandra on node 1 it fails and the log shows:

  INFO 03:23:25,284 Loading persisted ring state
  INFO 03:23:25,564 Starting Messaging Service on port 9000
  INFO 03:23:25,797 Handshaking version with sk.r252.0/10.2.252.0

ERROR 03:23:56,915 Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
 at 
org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
 at

org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
 at

org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
 at

org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
 at

org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
 at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
 at

org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
 at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
java.lang.RuntimeException: Unable to gossip with any seeds
 at 
org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
 at

org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
 at

org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
 at

org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
 at

org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
 at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
 at

org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
 at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
Exception encountered during startup: Unable to gossip with any seeds
ERROR 03:24:02,213 Exception in thread 
Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
 at

org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:349)
 at

org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:364)
 at

org.apache.cassandra.service.StorageService.access$000(StorageService.java:97)
 at

org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
 at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
 at java.lang.Thread.run(Unknown Source)

*bin/nodetool status*


Datacenter: 0
=
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address   Load   Owns (effective)  Host ID
 TokenRack
UN  10.0.2.1  35.8 KB100.0%
  0ae766db-

Query on Seed node

[Aravindan T]

 Hi ,

I  have a  4 node cassandra cluster with one node marked as seed node.  When i 
checked the data directory of seed node , it has two folders  
/keyspace/columnfamily.
But sstable db files are not available.the folder is empty.The db files are 
available in remaining nodes.

I want to know the reason why db files are not created in seed node ?
what will happen if all the nodes in a cluster is marked as seed node ??



Aravindan Thangavelu
Tata Consultancy Services
Mailto: aravinda...@tcs.com
Website: http://www.tcs.com

Experience certainty.   IT Services
Business Solutions
Consulting

=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you

Re: Query on Seed node

[Or Sher]

I'm guessing its just a coincident.. As far as I know, seeds have nothing
to do with where the data should be located.
I think there could be couple of reasons why you wouldn't see SSTables on a
specific column family folder, these are some of them:
- You're using a few distinct keys which non of them should be on that seed
node.
- Node hasn't flushed yet.. You can use nodetool flush to try and flush
memtables manually.
- You're using manual token assignment and you didn't not assign them well.


On Mon, Feb 3, 2014 at 1:25 PM, Aravindan T  wrote:

> Hi ,
>
> I have a  4 node cassandra cluster with one node marked as seed node. When
> i checked the data directory of seed node , it has two folders
> /keyspace/columnfamily.
> But sstable db files are not available.the folder is empty.The db files
> are available in remaining nodes.
>
> I want to know the reason why db files are not created in seed node ?
> what will happen if all the nodes in a cluster is marked as seed node ??
>
>
>
> Aravindan Thangavelu
> Tata Consultancy Services
> Mailto: aravinda...@tcs.com
> Website: http://www.tcs.com
> 
> Experience certainty. IT Services
> Business Solutions
> Consulting
> 
>
> =-=-=
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>


-- 
Or Sher

Re: Update cassandra from perl 5.8

[Jacob Rhoden]

Hi Paul,

I think some of the dependencies, like Snappy also require perl 5.10. If its 
not too difficult to get it to work in perl 5.8 it would be very much 
appreciated. If its a lot of work I wouldn’t want to waste your time, after all 
who still depends on perl 5.8? (:

(I can always resort to doing nightly dumps and transforms of the mysql 
database, but its no where near as good as having real time cassandra updates)

Thanks!
Jacob

On 3 Feb 2014, at 10:43 pm, Paul LeoNerd Evans  wrote:

> On Mon, 03 Feb 2014 19:44:47 +1100
> Jacob Rhoden  wrote:
> 
>> We’ve got a legacy application on Perl 5.8, don’t want to upgrade the
>> server for fear of something breaking. However for redundancy, It
>> would be great if I could update the app so it writes to both the
>> existing database, and cassandra. The Perl drivers appear to require
>> perl 5.10+
>> 
>>https://github.com/mkjellman/perlcassa
>>https://metacpan.org/pod/Net::Async::CassandraCQL
> 
> I wrote this latter.
> 
>> Just thought I would ask the list in the off chance that someone
>> might know of an alternate way to solve this problem or shoehorn a
>> perl driver onto this server.
> 
> If you want that to work on 5.8 I'm sure I could give it a go.
> 
> -- 
> Paul "LeoNerd" Evans
> 
> leon...@leonerd.org.uk
> ICQ# 4135350   |  Registered Linux# 179460
> http://www.leonerd.org.uk/

Balance Virtual Nodes

[Augusto Uehara]

Dear Cassandra list,

We are experiencing the following distibution of our tokens:

StateAddress Load Tokens   Owns Host ID 
   Rack
UN   10.180.199.188 108.72 GB 263 0.0% 
f2c84a20-a4b7-4db1-b5bd-95eaf231ac76 RAC1
UN   10.180.206.15 71.86 GB 254 25.0%   
a13afce3-7e98-4a3b-98d1-5bf6742435a4 RAC1
UN   10.45.163.69 152.23 GB 258 25.0%   
90247170-e2bc-43ae-a827-2dcb082e43fcRAC1
UN   10.235.36.125   103.91 GB 248 50.0%   
89f362f5-e844-470d-a303-f54e46abc1cb  RAC1

our num_tokens is set to 256 and this node started as a regular node and later 
this was migrated to a virtual node.

We are using the org.apache.cassandra.dht.Murmur3Partitioner.

I've already tried the move and the shuffle nodetool commands after reading a 
few other posts about this somewhere else, but with no luck.

Can anyone please advice? At this point, our servers don't have enough data 
that I wouldn't consider creating another cluster, swap the apps to the new 
cluster and migrate the data using snapshots or scripts, but I think this is a 
dumb way to do this...


Kind regards,
Augusto

[RELEASE] Apache Cassandra 1.2.14 released

[Sylvain Lebresne]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 1.2.14.

Cassandra is a highly scalable second-generation distributed database,
bringing together Dynamo's fully distributed design and Bigtable's
ColumnFamily-based data model. You can read more here:

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a maintenance/bug fix release[1] on the 1.2 series. As
always,
please pay attention to the release notes[2] and Let us know[3] if you were
to
encounter any problem.

Enjoy!

[1]: http://goo.gl/HVzqeg (CHANGES.txt)
[2]: http://goo.gl/967OAv (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: Automatic tombstone compaction

[Keith Wright]

Hi all,

I am running C* 1.2.13 with Vnodes at around 1 TB / node.  I just
noticed that one of my larger LCS CFs (300-400 GB/Node) is showing a
droppable tombstone ration of between 23 and 28% on my nodes.  I did not
indicate a value to be used in my table creation so I assume its using the
default 20%.  Note that I utilize TTLs somewhat heavily on this table with
most between 7 and 28 days against CQL3 collections (maps to be specific).
Couple of questions:

- Given compactions are not falling behind on my nodes, I would have
expected my ratio to be much closer to the default value of 20%.  Have
others seen this behavior?

- Again given that compactions are not falling behind, I am always up for
recovering data space and that are interested in lowering the default
value of 20%.  The documentation suggests that it will not trigger a
compaction for tombstones unless there are no compactions queued so I
assume this would not have a negative impact?

Thanks!

On 8/21/13, 11:23 AM, "Yuki Morishita"  wrote:

>Tamas,
>
>If there are rows with the same key in other SSTables, that rows won't
>be deleted.
>Tombstone compaction make guess if it can actually drop "safely" by
>scanning overlap with other SSTables.
>Do you have many rows in your large SSTable?
>If you don't, then chance to run tombstone compaction may be low.
>
>
>On Wed, Aug 21, 2013 at 9:33 AM,  
>wrote:
>> Hi,
>>
>>
>>
>> I ran upgradesstables as part of the Cassandra upgrade, before issuing
>>the
>> CQL alter command.
>>
>> According to the docs, SizeTieredCompactionStrategy is fine (that is
>>what I
>> used, and plan on continue using), and automatic tombstone compaction is
>> available for it:
>>
>> 
>>http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html#ca
>>ssandra/operations/ops_about_config_compact_c.html
>>
>> I just had to include the Œclass¹ in the alter statement, otherwise it
>>would
>> not accept my command.
>>
>> Is that not right?
>>
>>
>>
>> Thanks,
>>
>> Tamas
>>
>>
>>
>> From: Haithem Jarraya [mailto:a-hjarr...@expedia.com]
>> Sent: 21. august 2013 16:24
>> To: user@cassandra.apache.org
>> Subject: Re: Automatic tombstone compaction
>>
>>
>>
>> Hi,
>>
>>
>>
>> do you mean LeveledCompactionStrategy?
>>
>>
>>
>> Also you will need to run nodetool upgradesstables  [keyspace][cf_name]
>> after changing the compaction strategy.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Haithem Jarraya
>>
>> On 21 Aug 2013, at 15:15, tamas.fold...@thomsonreuters.com wrote:
>>
>>
>>
>> Hi,
>>
>>
>>
>> After upgrading from 1.0 to 1.2, I wanted to make use of the automatic
>> tombstone compaction feature, so using CQL3 I issued:
>>
>>
>>
>> ALTER TABLE versions WITH compaction = {'class' :
>> 'SizeTieredCompactionStrategy', 'min_threshold' : 4, 'max_threshold' :
>>32,
>> 'tombstone_compaction_interval' : 1, 'tombstone_threshold' : '0.1'};
>>
>>
>>
>> But I still see no trace that would suggest this works ­ we had 60G of
>>data
>> with TTL=1week pushed a while ago to the test cluster, the majority of
>>it
>> should be expired & compacted away by now. Not sure if it is relevant,
>>but
>> this old data is in one ~60G file + I have a few smaller files with
>>latest
>> data in them.
>>
>> Looking at JMX: DroppableTombstoneRatio = 0.892076544, which seems to
>>back
>> my theory.
>>
>> Am I doing something wrong, or am I expecting the wrong thing?
>>
>>
>>
>> Thanks,
>>
>> Tamas
>>
>>
>>
>>
>
>
>
>-- 
>Yuki Morishita
> t:yukim (http://twitter.com/yukim)

Re: Intermittent long application pauses on nodes

[Frank Ng]

I was able to send SafePointStatistics to another log file via the
additional JVM flags and recently noticed a pause of 9.3936600 seconds.
Here are the log entries:

GC Log file:
---
2014-01-31T07:49:14.755-0500: 137460.842: Total time for which application
threads were stopped: 0.1095540 seconds
2014-01-31T07:51:01.870-0500: 137567.957: Total time for which application
threads were stopped: 9.3936600 seconds
2014-01-31T07:51:02.537-0500: 137568.623: Total time for which application
threads were stopped: 0.1207440 seconds

JVM Stdout Log File:
---
 vmop [threads: total initially_running
wait_to_block][time: spin block sync cleanup vmop] page_trap_count
137460.734: GenCollectForAllocation  [ 421
00   ][ 0 0 23  0 84 ]  0
 vmop [threads: total initially_running
wait_to_block][time: spin block sync cleanup vmop] page_trap_count
137558.562: RevokeBias   [ 462
29   ][13 0   9391  1  0 ]  0



 vmop [threads: total initially_running
wait_to_block][time: spin block sync cleanup vmop] page_trap_count
137568.500: Deoptimize   [ 481
15   ][ 0 0118  0  1 ]  0
 vmop [threads: total initially_running
wait_to_block][time: spin block sync cleanup vmop] page_trap_count
137569.625: no vm operation  [ 483
01   ][ 0 0 18  0  0 ]  0
 vmop [threads: total initially_running
wait_to_block][time: spin block sync cleanup vmop] page_trap_count
137571.641: no vm operation  [ 483
01   ][ 0 0 42  1  0 ]  0
 vmop [threads: total initially_running
wait_to_block][time: spin block sync cleanup vmop] page_trap_count
137575.703: no vm operation  [ 483
01   ][ 0 0 25  1  0 ]  0

If SafepointStatistics are printed before the Application Stop times, then
it seems that the RevokeBias was the cause of the pause.
If SafepointStatistics are printed after the Application Stop times, then
it seems that the Deoptimize was the cause of the pause.
In addition, I see a strange dependency failed error relating to JMX in the
JVM stdout log file.

thanks


On Wed, Jan 29, 2014 at 4:44 PM, Benedict Elliott Smith <
belliottsm...@datastax.com> wrote:

> Add some more flags: -XX:+UnlockDiagnosticVMOptions -XX:LogFile=${path}
> -XX:+LogVMOutput
>
> I never figured out what kills stdout for C*. It's a library we depend on,
> didn't try too hard to figure out which one.
>
>
> On 29 January 2014 21:07, Frank Ng  wrote:
>
>> Benedict,
>> Thanks for the advice.  I've tried turning on PrintSafepointStatistics.
>> However, that info is only sent to the STDOUT console.  The cassandra
>> startup script closes the STDOUT when it finishes, so nothing is shown for
>> safepoint statistics once it's done starting up.  Do you know how to
>> startup cassandra and send all stdout to a log file and tell cassandra not
>> to close stdout?
>>
>> Also, we have swap turned off as recommended.
>>
>> thanks
>>
>>
>> On Wed, Jan 29, 2014 at 3:39 PM, Benedict Elliott Smith <
>> belliottsm...@datastax.com> wrote:
>>
>>> Frank,
>>>
>>>
>>> The same advice for investigating holds: add the VM flags 
>>> -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1   (you 
>>> could put something above 1 there, to reduce the amount of logging, since a 
>>> pause of 52s will be pretty obvious even if aggregated with lots of other 
>>> safe points; the count is the number of safepoints to aggregate into one 
>>> log message)
>>>
>>>
>>> 52s is a very extreme pause, and I would be surprised if revoke bias could 
>>> cause this. I wonder if the VM is swapping out.
>>>
>>>
>>>
>>> On 29 January 2014 19:02, Frank Ng  wrote:
>>>
 Thanks for the update.  Our logs indicated that there were 0 pending
 for CompactionManager at that time.  Also, there were no nodetool repairs
 running at that time.  The log statements above state that the application
 had to stop to reach a safepoint.  Yet, it doesn't say what is requesting
 the safepoint.


 On Wed, Jan 29, 2014 at 1:20 PM, Shao-Chuan Wang <
 shaochuan.w...@bloomreach.com> wrote:

> We had similar latency spikes when pending compactions can't keep it
> up or repair/streaming taking too much cycles.
>
>
> On Wed, Jan 29, 2014 at 10:07 AM, Frank Ng  wrote:
>
>> All,
>>
>> We've been having intermittent long application pauses (version
>> 1.2.8) and not sure if it's a cassandra bug.  During these pauses, there
>> are dropped messages in the cassandra log file along with the node seeing
>> other nodes as d

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

[Yuki Morishita]

if you are using < 2.0.4, then you are hitting
https://issues.apache.org/jira/browse/CASSANDRA-6527


On Mon, Feb 3, 2014 at 2:51 AM, olek.stas...@gmail.com
 wrote:
> Hi All,
> We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via
> 1.2.10). Probably after upgradesstable  (but it's only a guess,
> because we noticed problem few weeks later), some rows became
> tombstoned. They just disappear from results of queries. After
> inverstigation I've noticed, that they are reachable via sstable2json.
> Example output for "non-existent" row:
>
> {"key": "6e6e37716c6d665f6f61695f6463","metadata": {"deletionInfo":
> {"markedForDeleteAt":2201170739199,"localDeletionTime":0}},"columns":
> [["DATA","3c6f61695f64633a64(...)",1357677928108]]}
> ]
>
> If I understand correctly row is marked as deleted with timestamp in
> the far future, but it's still on the disk. Also localDeletionTime is
> set to 0, which may means, that it's kind of internal bug, not effect
> of client error. So my question is: is it true, that upgradesstable
> may do soemthing like that? How to find reasons for such strange
> cassandra behaviour? Is there any option of recovering such strange
> marked nodes?
> This problem touches about 500K rows of all 14M in our database, so
> the percentage is quite big.
> best regards
> Aleksander
>
> 2013-12-12 Robert Coli :
>> On Wed, Dec 11, 2013 at 6:27 AM, Mathijs Vogelzang 
>> wrote:
>>>
>>> When I use sstable2json on the sstable on the destination cluster, it has
>>> "metadata": {"deletionInfo":
>>> {"markedForDeleteAt":1796952039620607,"localDeletionTime":0}}, whereas
>>> it doesn't have that in the source sstable.
>>> (Yes, this is a timestamp far into the future. All our hosts are
>>> properly synced through ntp).
>>
>>
>> This seems like a bug in sstableloader, I would report it on JIRA.
>>
>>>
>>> Naturally, copying the data again doesn't work to fix it, as the
>>> tombstone is far in the future. Apart from not having this happen at
>>> all, how can it be fixed?
>>
>>
>> Briefly, you'll want to purge that tombstone and then reload the data with a
>> reasonable timestamp.
>>
>> Dealing with rows with data (and tombstones) in the far future is described
>> in detail here :
>>
>> http://thelastpickle.com/blog/2011/12/15/Anatomy-of-a-Cassandra-Partition.html
>>
>> =Rob
>>



-- 
Yuki Morishita
 t:yukim (http://twitter.com/yukim)

Re: Intermittent long application pauses on nodes

[Benedict Elliott Smith]

Hi Frank,

The "9391" under RevokeBias is the number of milliseconds spent
synchronising on the safepoint prior to the VM operation, i.e. the time it
took to ensure all application threads were stopped. So this is the
culprit. Notice that the time spent spinning/blocking for the threads we
are supposed to be waiting on is very low; it looks to me that this is time
spent waiting for CMS threads to yield, though it is very hard to say with
absolute certainty. It doesn't look like the issue is actually the
RevokeBias itself, anyway.

I think we should take this off list. It definitely warrants a ticket,
though I expect this will be difficult to pin down, so you will have to be
willing to experiment a bit with us, but we would be very grateful for the
help. If you can pin down and share a specific workload that triggers this
we may be able to do it without you though!

It's possible that this is a JVM issue, but if so there may be some
remedial action we can take anyway. There are some more flags we should
add, but we can discuss that once you open a ticket. If you could include
the strange JMX error as well, that might be helpful.

Thanks,

Benedict


On 3 February 2014 15:34, Frank Ng  wrote:

> I was able to send SafePointStatistics to another log file via the
> additional JVM flags and recently noticed a pause of 9.3936600 seconds.
> Here are the log entries:
>
> GC Log file:
> ---
> 2014-01-31T07:49:14.755-0500: 137460.842: Total time for which application
> threads were stopped: 0.1095540 seconds
> 2014-01-31T07:51:01.870-0500: 137567.957: Total time for which application
> threads were stopped: 9.3936600 seconds
> 2014-01-31T07:51:02.537-0500: 137568.623: Total time for which application
> threads were stopped: 0.1207440 seconds
>
> JVM Stdout Log File:
> ---
>  vmop [threads: total initially_running
> wait_to_block][time: spin block sync cleanup vmop] page_trap_count
> 137460.734: GenCollectForAllocation  [ 421
> 00   ][ 0 0 23  0 84 ]  0
>  vmop [threads: total initially_running
> wait_to_block][time: spin block sync cleanup vmop] page_trap_count
> 137558.562: RevokeBias   [ 462
> 29   ][13 0   9391  1  0 ]  0
> 
>  ctxk='javax/management/ObjectName$Property'
> witness='javax/management/ObjectName$PatternProperty' stamp='137568.503'/>
> 
>  vmop [threads: total initially_running
> wait_to_block][time: spin block sync cleanup vmop] page_trap_count
> 137568.500: Deoptimize   [ 481
> 15   ][ 0 0118  0  1 ]  0
>  vmop [threads: total initially_running
> wait_to_block][time: spin block sync cleanup vmop] page_trap_count
> 137569.625: no vm operation  [ 483
> 01   ][ 0 0 18  0  0 ]  0
>  vmop [threads: total initially_running
> wait_to_block][time: spin block sync cleanup vmop] page_trap_count
> 137571.641: no vm operation  [ 483
> 01   ][ 0 0 42  1  0 ]  0
>  vmop [threads: total initially_running
> wait_to_block][time: spin block sync cleanup vmop] page_trap_count
> 137575.703: no vm operation  [ 483
> 01   ][ 0 0 25  1  0 ]  0
>
> If SafepointStatistics are printed before the Application Stop times, then
> it seems that the RevokeBias was the cause of the pause.
> If SafepointStatistics are printed after the Application Stop times, then
> it seems that the Deoptimize was the cause of the pause.
> In addition, I see a strange dependency failed error relating to JMX in
> the JVM stdout log file.
>
> thanks
>
>
> On Wed, Jan 29, 2014 at 4:44 PM, Benedict Elliott Smith <
> belliottsm...@datastax.com> wrote:
>
>> Add some more flags: -XX:+UnlockDiagnosticVMOptions -XX:LogFile=${path}
>> -XX:+LogVMOutput
>>
>> I never figured out what kills stdout for C*. It's a library we depend
>> on, didn't try too hard to figure out which one.
>>
>>
>> On 29 January 2014 21:07, Frank Ng  wrote:
>>
>>> Benedict,
>>> Thanks for the advice.  I've tried turning on PrintSafepointStatistics.
>>> However, that info is only sent to the STDOUT console.  The cassandra
>>> startup script closes the STDOUT when it finishes, so nothing is shown for
>>> safepoint statistics once it's done starting up.  Do you know how to
>>> startup cassandra and send all stdout to a log file and tell cassandra not
>>> to close stdout?
>>>
>>> Also, we have swap turned off as recommended.
>>>
>>> thanks
>>>
>>>
>>> On Wed, Jan 29, 2014 at 3:39 PM, Benedict Elliott Smith <
>>> belliottsm...@datastax.com> wrote:
>>>
 Frank,


 The same advice for investigating holds: add the VM 

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

[olek.stas...@gmail.com]

Ok, but will upgrade "resurrect" my data? Or maybe I should perform
additional action to bring my system to correct state?

best regards

Aleksander
3 lut 2014 17:08 "Yuki Morishita"  napisał(a):

> if you are using < 2.0.4, then you are hitting
> https://issues.apache.org/jira/browse/CASSANDRA-6527
>
>
> On Mon, Feb 3, 2014 at 2:51 AM, olek.stas...@gmail.com
>  wrote:
> > Hi All,
> > We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via
> > 1.2.10). Probably after upgradesstable  (but it's only a guess,
> > because we noticed problem few weeks later), some rows became
> > tombstoned. They just disappear from results of queries. After
> > inverstigation I've noticed, that they are reachable via sstable2json.
> > Example output for "non-existent" row:
> >
> > {"key": "6e6e37716c6d665f6f61695f6463","metadata": {"deletionInfo":
> > {"markedForDeleteAt":2201170739199,"localDeletionTime":0}},"columns":
> > [["DATA","3c6f61695f64633a64(...)",1357677928108]]}
> > ]
> >
> > If I understand correctly row is marked as deleted with timestamp in
> > the far future, but it's still on the disk. Also localDeletionTime is
> > set to 0, which may means, that it's kind of internal bug, not effect
> > of client error. So my question is: is it true, that upgradesstable
> > may do soemthing like that? How to find reasons for such strange
> > cassandra behaviour? Is there any option of recovering such strange
> > marked nodes?
> > This problem touches about 500K rows of all 14M in our database, so
> > the percentage is quite big.
> > best regards
> > Aleksander
> >
> > 2013-12-12 Robert Coli :
> >> On Wed, Dec 11, 2013 at 6:27 AM, Mathijs Vogelzang <
> math...@apptornado.com>
> >> wrote:
> >>>
> >>> When I use sstable2json on the sstable on the destination cluster, it
> has
> >>> "metadata": {"deletionInfo":
> >>> {"markedForDeleteAt":1796952039620607,"localDeletionTime":0}}, whereas
> >>> it doesn't have that in the source sstable.
> >>> (Yes, this is a timestamp far into the future. All our hosts are
> >>> properly synced through ntp).
> >>
> >>
> >> This seems like a bug in sstableloader, I would report it on JIRA.
> >>
> >>>
> >>> Naturally, copying the data again doesn't work to fix it, as the
> >>> tombstone is far in the future. Apart from not having this happen at
> >>> all, how can it be fixed?
> >>
> >>
> >> Briefly, you'll want to purge that tombstone and then reload the data
> with a
> >> reasonable timestamp.
> >>
> >> Dealing with rows with data (and tombstones) in the far future is
> described
> >> in detail here :
> >>
> >>
> http://thelastpickle.com/blog/2011/12/15/Anatomy-of-a-Cassandra-Partition.html
> >>
> >> =Rob
> >>
>
>
>
> --
> Yuki Morishita
>  t:yukim (http://twitter.com/yukim)
>

Bulk writes and key cache

[Daning Wang]

Does Cassandra put keys in key cache during the write path?

If I have two tables, the key cache for the first table was warmed up
nicely, and I want to insert millions rows in the second table, and there
is no read on the second table yet, will that affect cache hit ratio for
the first table?

Thanks,

Daning

Cassandra 2.0 with Hadoop 2.x?

[Clint Kelly]

Folks,

Has anyone out there used Cassandra 2.0 with Hadoop 2.x?  I saw this
discussion on the Cassandra JIRA:

https://issues.apache.org/jira/browse/CASSANDRA-5201

but the fix referenced
(https://github.com/michaelsembwever/cassandra-hadoop) is for
Cassandra 1.2.

I put together a similar patch for Cassandra 2.0 for anyone who is interested:

https://github.com/wibiclint/cassandra2-hadoop2

but I'm wondering if there is a more official solution to this
problem.  Any help would be appreciated.  Thanks!

Best regards,
Clint

Re: Bulk writes and key cache

[Robert Coli]

On Mon, Feb 3, 2014 at 10:03 AM, Daning Wang  wrote:

> Does Cassandra put keys in key cache during the write path?
>

No.

=Rob

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

[Robert Coli]

On Mon, Feb 3, 2014 at 12:51 AM, olek.stas...@gmail.com <
olek.stas...@gmail.com> wrote:

> We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via
> 1.2.10). Probably after upgradesstable  (but it's only a guess,
> because we noticed problem few weeks later), some rows became
> tombstoned.


To be clear, you didn't run SSTableloader at all? If so, this is the
hypothetical case where normal streaming operations (replacing a node? what
streaming did you do?) results in data loss...

Also, CASSANDRA-6527 is a good reminder regarding the following :

https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/

=Rob

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

[olek.stas...@gmail.com]

Yes, I haven't run sstableloader. The data loss apperared somwhere on the line:
1.1.7-> 1.2.10 -> upgradesstable -> 2.0.2 -> normal operations ->2.0.3
normal operations -> now
Today I've noticed that oldest files with broken values appear during
repair (we do repair once a week on each node). Maybe it's the repair
operation, which caused data loss? I've no idea. Currently our cluster
is runing 2.0.3 version.
We can do some tests on data to give you all info to track the bug.
But our most crucial question is: can we recover loss, or should we
start to think how to re-gather them?
best regards
Aleksander
ps. I like your link Rob, i'll pin it over my desk ;) In Oracle there
were a rule: never deploy RDBMS before release 2 ;)

2014-02-03 Robert Coli :
> On Mon, Feb 3, 2014 at 12:51 AM, olek.stas...@gmail.com
>  wrote:
>>
>> We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via
>> 1.2.10). Probably after upgradesstable  (but it's only a guess,
>> because we noticed problem few weeks later), some rows became
>> tombstoned.
>
>
> To be clear, you didn't run SSTableloader at all? If so, this is the
> hypothetical case where normal streaming operations (replacing a node? what
> streaming did you do?) results in data loss...
>
> Also, CASSANDRA-6527 is a good reminder regarding the following :
>
> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>
> =Rob

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

[Robert Coli]

On Mon, Feb 3, 2014 at 1:02 PM, olek.stas...@gmail.com <
olek.stas...@gmail.com> wrote:

> Today I've noticed that oldest files with broken values appear during
> repair (we do repair once a week on each node). Maybe it's the repair
> operation, which caused data loss?


Yes, unless you added or removed or replaced nodes, it would have to be the
repair operation, which streams SSTables. Did you run the repair during the
upgradesstables?


> I've no idea. Currently our cluster
> is runing 2.0.3 version.
>

2.0.3 has serious bugs, upgrade to 2.0.4 ASAP.


> But our most crucial question is: can we recover loss, or should we
> start to think how to re-gather them?
>

If I were you, I would do the latter. You can to some extent recover them
via manual processes dumping with sstable2json and so forth, but it will be
quite painful.

http://thelastpickle.com/2011/12/15/Anatomy-of-a-Cassandra-Partition/

Contains an explanation of how one could deal with it.

=Rob




> best regards
> Aleksander
> ps. I like your link Rob, i'll pin it over my desk ;) In Oracle there
> were a rule: never deploy RDBMS before release 2 ;)
>
> 2014-02-03 Robert Coli :
> > On Mon, Feb 3, 2014 at 12:51 AM, olek.stas...@gmail.com
> >  wrote:
> >>
> >> We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via
> >> 1.2.10). Probably after upgradesstable  (but it's only a guess,
> >> because we noticed problem few weeks later), some rows became
> >> tombstoned.
> >
> >
> > To be clear, you didn't run SSTableloader at all? If so, this is the
> > hypothetical case where normal streaming operations (replacing a node?
> what
> > streaming did you do?) results in data loss...
> >
> > Also, CASSANDRA-6527 is a good reminder regarding the following :
> >
> >
> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
> >
> > =Rob
>

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

[olek.stas...@gmail.com]

2014-02-03 Robert Coli :
> On Mon, Feb 3, 2014 at 1:02 PM, olek.stas...@gmail.com
>  wrote:
>>
>> Today I've noticed that oldest files with broken values appear during
>> repair (we do repair once a week on each node). Maybe it's the repair
>> operation, which caused data loss?
>
>
> Yes, unless you added or removed or replaced nodes, it would have to be the
> repair operation, which streams SSTables. Did you run the repair during the
> upgradesstables?

No, i've done repair after upgrade sstables. In fact it was about 4
weeks after, because of bug:
https://issues.apache.org/jira/browse/CASSANDRA-6277. We upgrded cass
to 2.0.2 and then after ca 1 month to 2.0.3 because of 6277. Then we
were able to do repair, so I set up cron to do it weekly on each node.
(it was about 10 dec 2013) the loss was discovered about new year's
eve.

>
>>
>> I've no idea. Currently our cluster
>> is runing 2.0.3 version.
>
>
> 2.0.3 has serious bugs, upgrade to 2.0.4 ASAP.
OK
>
>>
>> But our most crucial question is: can we recover loss, or should we
>> start to think how to re-gather them?
>
>
> If I were you, I would do the latter. You can to some extent recover them
> via manual processes dumping with sstable2json and so forth, but it will be
> quite painful.
>
> http://thelastpickle.com/2011/12/15/Anatomy-of-a-Cassandra-Partition/
>
> Contains an explanation of how one could deal with it.
Sorry, but I have to admit, that i can't transfer this solution to my
problem. Could you briefly describe steps I should perform to recover?
best regards
Aleksander

>
> =Rob
>
>
>
>>
>> best regards
>> Aleksander
>> ps. I like your link Rob, i'll pin it over my desk ;) In Oracle there
>> were a rule: never deploy RDBMS before release 2 ;)
>>
>> 2014-02-03 Robert Coli :
>> > On Mon, Feb 3, 2014 at 12:51 AM, olek.stas...@gmail.com
>> >  wrote:
>> >>
>> >> We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via
>> >> 1.2.10). Probably after upgradesstable  (but it's only a guess,
>> >> because we noticed problem few weeks later), some rows became
>> >> tombstoned.
>> >
>> >
>> > To be clear, you didn't run SSTableloader at all? If so, this is the
>> > hypothetical case where normal streaming operations (replacing a node?
>> > what
>> > streaming did you do?) results in data loss...
>> >
>> > Also, CASSANDRA-6527 is a good reminder regarding the following :
>> >
>> >
>> > https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>> >
>> > =Rob
>
>

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

[Robert Coli]

On Mon, Feb 3, 2014 at 2:17 PM, olek.stas...@gmail.com <
olek.stas...@gmail.com> wrote:

> No, i've done repair after upgrade sstables. In fact it was about 4
> weeks after, because of bug:
>

If you only did a repair after you upgraded SSTables, when did you have an
opportunity to hit :

https://issues.apache.org/jira/browse/CASSANDRA-6527

... which relies on you having multiple versions of SStables while
streaming?

Did you do any operation which involves streaming? (Add/Remove/Replace a
node?)

=Rob

Re: Intermittent long application pauses on nodes

[Robert Coli]

On Mon, Feb 3, 2014 at 8:52 AM, Benedict Elliott Smith <
belliottsm...@datastax.com> wrote:

>
> It's possible that this is a JVM issue, but if so there may be some
> remedial action we can take anyway. There are some more flags we should
> add, but we can discuss that once you open a ticket. If you could include
> the strange JMX error as well, that might be helpful.
>

It would be appreciated if you could inform this thread of the JIRA ticket
number, for the benefit of the community and google searchers. :)

=Rob

socket timeout errors in one DC in a multi DC cluster

[Senthil, Athinanthny X. -ND]

Experiencing socket timeout errors in one  DC in most of the nodes in multi dc 
cluster. Here is error. Client is having intermittent high response time issues 
in this DC. DC1 does not experience any timeout issues, but DC2 does 
though.  This error started occurring recently and repeats consecutive days. 
Any suggestions on cause for it?

During this situation, when we try to run queries on CQL3 on local server 
itself, sometime we get rpc_timeout errors. But this is intermittent as well.


ERROR [Thrift:97]   CustomTThreadPoolServer.java (line 219) Error occurred 
during processing of message.
com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 2 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
at org.apache.cassandra.service.ClientState.authorize(ClientState.java:308)
at 
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:178)
at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:171)
at 
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:155)
at 
org.apache.cassandra.thrift.CassandraServer.createMutationList(CassandraServer.java:681)
at 
org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:749)
at com.datastax.bdp.server.DseServer.batch_mutate(DseServer.java:931)
at 
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3622)
at 
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3610)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at 
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Re: One of my nodes is in the wrong datacenter - help!

[Robert Coli]

On Sun, Feb 2, 2014 at 10:48 AM, Sholes, Joshua <
joshua_sho...@cable.comcast.com> wrote:

>  I had a node in my 8-node production 1.2.8 cluster have a serious
> problem and need to be removed and rebuilt.   However, after doing nodetool
> removenode and then bootstrapping a new node on the same IP address, the
> new node somehow ended up with a different datacenter name (the rest of the
> nodes are in dc $NAME, and the new one is in dc $NAME6934724 -- as in, a
> string of seemingly random numbers appended to the correct name).   How can
> I force it to change DC names back to what it should be?
>

You could change the entry in the system.local columnfamily on the affected
node...

cqlsh > update system.local set data_center = "$NAME";

... but that is Not Supported and may have side effects of which I am not
aware.

 I'm working with 500+GB per node here so bootstrapping it again is not a
> huge issue, but I'd prefer to avoid it anyway.  I am NOT able to change the
> node's IP address at this time so I'm stuck with bootstrapping a new node
> in the same place, which my gut feeling tells me might be part of the
> problem.
>

Note that replace_node/replace_token are broken in 1.2.8, did you attempt
to use either of these? I presume not because you said you did removenode...

 If I were you, I would probably removenode and re-bootstrap, as the safest
alternative.

As an aside, while trying to deal with this issue you should be aware of
this ticket, so you do not do the sequence of actions it describes.

https://issues.apache.org/jira/browse/CASSANDRA-6615

=Rob