RE: lots of DigestMismatchException in cassandra3

That's a debug message. From the sound of it, it's triggered on read where there is a digest mismatch between replicas. As to whether it's normal, well that depends on your cluster. Are the nodes reporting lots of dropped mutations and are you writing at

Re: lots of DigestMismatchException in cassandra3

Actually, just saw the error message in those logs and what you're looking at is probably https://issues.apache.org/jira/browse/CASSANDRA-12694 Kurt Greaves k...@instaclustr.com www.instaclustr.com On 21 November 2016 at 08:59, kurt Greaves wrote: > That's a debug message. From the sound of i

Out of memory and/or OOM kill on a cluster

Hello, we have a 8 node Cassandra 2.1.15 cluster at work which is giving us a lot of trouble lately. The problem is simple: nodes regularly die because of an out of memory exception or the Linux OOM killer decides to kill the process. For a couple of weeks now we increased the heap to 20Gb hop

NoHostAvailableException

Following exception intermittently thrown by datastax java driver though all nodes are up.(Happening for both read & write queries) *"Exception com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) at com.datastax.driver.core.exception

Re: NoHostAvailableException

Try to set broadcast_rpc_address on each node to its real external IP address. Best regards, Vladimir Yudovin, Winguzone - Hosted Cloud Cassandra Launch your cluster in minutes. On Mon, 21 Nov 2016 05:47:00 -0500techpyaasa . wrote Following exception

Re: Out of memory and/or OOM kill on a cluster

Did you try any value in the range 8-20 (e.g. 60-70% of physical memory). Also how many tables do you have across all keyspaces? Each table can consume minimum 1M of Java heap. Best regards, Vladimir Yudovin, Winguzone - Hosted Cloud Cassandra Launch your cluster in minutes. On Mon

Re: Out of memory and/or OOM kill on a cluster

Hi Vincent, one of the usual causes of OOMs is very large partitions. Could you check your nodetool cfstats output in search of large partitions ? If you find one (or more), run nodetool cfhistograms on those tables to get a view of the partition sizes distribution. Thanks On Mon, Nov 21, 2016 a

Re: NoHostAvailableException

Hi Vladimir, I have not modified anything for broadcasr_address, I left as it was.. *# Leaving this blank will set it to the same value as listen_address* *# broadcast_address: 1.2.3.4* So the comment above says "*Leaving this blank will set it to the same value as listen_address" *, so it shud

Re: NoHostAvailableException

Not broadcast_address, but broadcast_rpc_address (you gave this example:rpc_address: 0.0.0.0 , broadcast_rpc_address: 1.2.3.4) Best regards, Vladimir Yudovin, Winguzone - Hosted Cloud Cassandra Launch your cluster in minutes. On Mon, 21 Nov 2016 08:14:38 -0500techpyaasa .

Re: NoHostAvailableException

Sorry it was typo.. It is *broadcast_address and not **broadcast_*rpc*_address.* And also there is no such configuration in cass.yaml with *broadcast_rpc_address *in c*-2.0.17. Very sorry once again. This is configrn I have in cass.yaml *listen_address: [external IP]* *# Address to broadcast to

Re: Data migration from Oracle to Cassandra

Hi Shashidhar, I have done something like that at reasonably high scale, migrating a few billions of Oracle records to Cassandra. Basically the process we used is : the app will perform the write in cassandra for new or updated records, the batch will backfeed old data in cassandra. We wrote eve

Re: NoHostAvailableException

Yaml in 2.0.17 says # The address to bind the Thrift RPC service and native transport # server -- clients connect here. # # Leaving this blank has the same effect it does for ListenAddress, # (i.e. it will be based on the configured hostname of the node). # # Note that unlike ListenAddress a

Re: Out of memory and/or OOM kill on a cluster

@Vladimir We tried with 12Gb and 16Gb, the problem appeared eventually too. In this particular cluster we have 143 tables across 2 keyspaces. @Alexander We have one table with a max partition of 2.68GB, one of 256 MB, a bunch with the size varying between 10MB to 100MB ~. Then there's the

Re: Out of memory and/or OOM kill on a cluster

Vincent, only the 2.68GB partition is out of bounds here, all the others (<256MB) shouldn't be much of a problem. It could put pressure on your heap if it is often read and/or compacted. But to answer your question about the 1% harming the cluster, a few big partitions can definitely be a big prob

Re: Node replacement failed in 2.2

Hmm, I don't think we use join_ring=false or write_survey=true for that node. I already remove_node to take the bad node out of ring, will try to have more debug logs next time. Thanks. On Sun, Nov 20, 2016 at 2:31 PM, Paulo Motta wrote: > Is there any chance the replaced node recently resumed

Is it *safe* to issue multiple replace-node at the same time?

Hi guys, Sometimes we need to replace multiple hosts in the same rack, is it safe to replace them in parallel, using the replace-node command? Will it cause any data inconsistency if we do so? Thanks Dikang. -- Dikang

Nodetool Repair Best Practices

Hey everyone, We've just migrated to a new Cassandra cluster running 3.7 and wanted to get some information on best practices when running nodetool repair; our last cluster was 1.2 and per the documentation it seems that a lot of behavior has changed between those versions. >From a read of the do

Re: Out of memory and/or OOM kill on a cluster

Thanks for your answer Alexander. We're writing constantly to the table, we estimate it's something like 1.5k to 2k writes per second. Some of these requests update a bunch of fields, some update fields + append something to a set. We don't read constantly from it but when we do it's a lot of re

Re: NoHostAvailableException

Hi Vladimir, I have attached cassandra.yaml we have in our setup, please check once. - do you have native port 9042 open in firewall ? Yes, 9042 is opened on our firewall, checked with our team - Can you connect to cluster with cqlsh? Yes, Im able to connect cluster using cqlsh. What else could

Re: NoHostAvailableException

Hi, as I mentioned about rpc_address: 0.0.0.0 YAML says it is allowed to specify 0.0.0.0 ... but that will break clients that rely on node auto-discovery. Try set rpc_address: external_ip Best regards, Vladimir Yudovin, Winguzone - Hosted Cloud Cassandra Launch your cluster in minutes.

Re: Is it *safe* to issue multiple replace-node at the same time?

Same rack and no range movements, my first instinct is to say yes it is safe (I like to treat racks as one giant meta node). However I would want to have a read through the replace code. On Mon, Nov 21, 2016, 07:22 Dikang Gu wrote: > Hi guys, > > Sometimes we need to replace multiple hosts in th

Cassandra Encryption

Hello, I am setting up encryption on one of my cassandra cluster using the below procedure. server_encryption_options: internode_encryption: all keystore: /etc/keystore keystore_password: x truststore: /etc/truststore truststore_password: x http://docs.oracle.com/java

Re: Is it *safe* to issue multiple replace-node at the same time?

On 21 November 2016 at 18:58, Ben Bromhead wrote: > Same rack and no range movements, my first instinct is to say yes it is > safe (I like to treat racks as one giant meta node). However I would want > to have a read through the replace code. This is assuming RF<=# of racks as well (and NTS).

Re: Cassandra reaper

Hello Alexander, Thanks for the help, I couldn't get around with my issue. but I started using : https://github.com/thelastpickle/cassandra-reaper it works like a charm :) I am using GUI, I just need to tweak/play with the configuration. Thanks again for the help On Tue, Nov 1, 2016 at 12:26 P

Re: data not replicated on new node

Hello Shalom, Vladimir, Thanks for your help. I had initially 3 nodes, had a hardware failure and reinstalled Cassandra on the node (I took that opportunity to upgrade from 3.1.1 to 3.0.9). I ran nodetool upgradesstables and nodetool repair on each node once I updated Cassandra. The 3 nodes

Re: Cassandra reaper

I noticed that I am not able to add schedules, but I can run repairs. Is there some limitation on the opensource for adding the schedules? On Mon, Nov 21, 2016 at 4:25 PM, Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote: > Hello Alexander, > > Thanks for the help, I couldn't get around

Re: Clarify Support for 2.2 on Download Page

Hi Derek You should subscribe and post this question to the Dev list, they will be able to get you sorted quickly! Normally you can edit documentation directly via github (e.g. https://github.com/apache/cassandra/tree/trunk/doc/source), however the download source appears to be outside the Cassan

Re: Cassandra Encryption

Hi Jai, so do you generate separate certificate for each node? Why not use one certificate for all nodes? Best regards, Vladimir Yudovin, Winguzone - Hosted Cloud Cassandra Launch your cluster in minutes. On Mon, 21 Nov 2016 17:25:11 -0500Jai Bheemsen Rao Dhanwada

Re: Cassandra reaper

Hi Jai, Reaper is fully open sourced and you should be able to add schedules. Could you open an issue on GitHub and provide both configuration and error output (if any) ? >> https://github.com/thelastpickle/cassandra-reaper/issues Thanks, On Tue, Nov 22, 2016 at 1:59 AM Jai Bheemsen Rao Dhanwad

RE: lots of DigestMismatchException in cassandra3

Hi Kurt, Thank you for the information, but the error “Corrupt empty row found in unfiltered partition” seems not related to the “Mismatch”; the time they occurred didn’t match. We use “QUORUM” consistency level for both read and write and I didn’t notice any failed writing in the log. Any other

Re: data not replicated on new node

*I took that opportunity to upgrade from 3.1.1 to 3.0.9* If my guess is right and you meant that you upgraded from 2.1.1 to 3.0.9 directly, then this might cause some issues (not necessarily the issue at hand though). The proper upgrade process should be to 2.1.9 and from there upgrade to 3.0.x.

Re: Cassandra reaper

Done, thanks https://github.com/thelastpickle/cassandra-reaper/issues/22 On Mon, Nov 21, 2016 at 9:53 PM, Alexander Dejanovski < a...@thelastpickle.com> wrote: > Hi Jai, > > Reaper is fully open sourced and you should be able to add schedules. > Could you open an issue on GitHub and provide both