Cassandra DC2 nodes down after increasing write requests on DC1 nodes

Hello, We are using Cassandra 2.1.2 in a multi dc cluster (30 servers on DC1 and 10 on DC2) with a key space replication factor of 1 on DC1 and 2 on DC2. For some reason when we increase the volume of write requests on DC1 (using ONE or LOCAL_ONE), the Cassandra java process on DC2 nodes goes dow

Re: Two writers appending to a set to see which one wins?

You may be able to do something with conditional updates, however trying to use Cassandra for this kind of coordination smells to me a lot like typical antipatterns (eg write then read or read then write). You probably would do better if you need one writer to consistently win a race condition ove

Re: Reading the write time of each value in a set?

I'm not aware of a way to query TTL or writetime on collections from CQL yet. You can access this information from Thrift though. On Sat Nov 15 2014 at 12:51:55 AM DuyHai Doan wrote: > Why don't you use map to store write time as value and data as key? > Le 15 nov. 2014 00:24, "Kevin Burton" a

Re: Cassandra DC2 nodes down after increasing write requests on DC1 nodes

> load average on DC1 nodes are around 3-5 and on DC2 around 7-10 Anecdotally I can say that loads in the 7-10 range have been dangerously high. When we had a cluster running in this range, the cluster was falling behind on important tasks such as compaction, and we really struggled to successful

Re: Cassandra DC2 nodes down after increasing write requests on DC1 nodes

Hi Eric, Thanks for your reply. I said that load was not a big deal, because ops center shows this loads as green, not as yellow or red at all. Also, our servers have many processors/threads, so I *think* this load is not problematic. My assumption is that for some reason the DC2 10 nodes are

Problem with performance, memory consumption, and RLIMIT_MEMLOCK

Hi, I have a very simple table in cassandra that contains only three columns: id, time and blob with data. I added 1M rows of data and now the database is about 12GB on disk. 1M is only part of data I want to store in the database, it's necessary to synchronize this table with external source. In o

What time range does nodetool cfhistograms use?

Hi all, Over what time range does "nodetool cfhistograms" operate? I am using Cassandra 2.0.8.39. I am trying to debug some very high 95th and 99th percentile read latencies in an application that I'm working on. I tried running nodetool cfhistograms to get a flavor for the distribution of read

Re: What time range does nodetool cfhistograms use?

Hi Clint, The values of SSTables, Write Latency and Read Latency will be reset on node start/restart and after running the cfhistograms command itself. The values of Row Size and Column Count are calculated at startup and then re-evaluated during compaction. Mark On 16 November 2014 17:12, Cl

Re: Cassandra DC2 nodes down after increasing write requests on DC1 nodes

Hello Gabriel, On Sun, Nov 16, 2014 at 7:25 AM, Gabriel Menegatti wrote: > I said that load was not a big deal, because ops center shows this loads as > green, not as yellow or red at all. > > Also, our servers have many processors/threads, so I *think* this load is > not problematic. I've seen

Re: Reading the write time of each value in a set?

Thanks. I’ll probably file a bug for this, assuming one doesn’t already exist. On Sun, Nov 16, 2014 at 6:20 AM, Eric Stevens wrote: > I'm not aware of a way to query TTL or writetime on collections from CQL > yet. You can access this information from Thrift though. > > On Sat Nov 15 2014 at 12:

Cassandra 2.1.1 Out of Memory Errors

I'm running a 4 node cluster with RF=3, CL of QUORUM for writes and ONE for reads. Each node has 3.7GB RAM with 32GB SSD HD, commitlog is on another HD. Currently each node has about 12GB of data. Cluster is always normal unless repair happens, that's when some nodes go to medium health in term

Re: Cassandra 2.1.1 Out of Memory Errors

"I do use rows that can span few thousands to a few million, I'm not sure if range slices happen using the memory." What are your query patterns ? For a given partition, take a slice of xxx columns ? Or give me a range of partitions ? For the 1st scenario, depending on how many columns you want

Re: What time range does nodetool cfhistograms use?

Thanks, Mark! A couple of other questions about the command: * Does it show statistics for just the current node on which I am running, or for the entire cluster? * Are the read latencies shown the latencies within a single host, or are they the end-to-end latencies from the coordinator node? -C

Best practices for route tracing

Hi all, I am trying to debug some high-latency outliers (99th percentile) in an application I'm working on. I thought that I could turn on route tracing, print the route traces to logs, and then examine my logs after a load test to find the highest-latency paths and figure out what is going on.

Re: What time range does nodetool cfhistograms use?

"Does it show statistics for just the current node on which I am running, or for the entire cluster?" --> only current node. "Are the read latencies shown the latencies within a single host, or are they the end-to-end latencies from the coordinator node?" --> cfhistograms shows metrics at table/n

Re: What time range does nodetool cfhistograms use?

Thanks for the quick reply! I completely forgot about proxyhistograms! On Sun, Nov 16, 2014 at 1:56 PM, DuyHai Doan wrote: > "Does it show statistics for just the current node on which I am running, > or for the entire cluster?" --> only current node. > > "Are the read latencies shown the laten

Re: Cassandra 2.1.1 Out of Memory Errors

DuyHai Doan gmail.com> writes: > > > "I do use rows that can span few thousands to a few million, I'm not sure if rangeslices happen using the memory." > What are your query patterns ? For a given partition, take a slice of xxx columns ? Or give me a range of partitions ? > >  For the 1st sc

read repair across DC and latency

I have a CF that use the default, read_repair_chance (0.1) and dc_read_repair_chance(0). Our read and write is all local_quorum, on one of the 2 DC, replication of 3. so a read will have 10% chance trigger a read repair to other DC. # I have read that read repair suppose to be running as back

Re: Cassandra 2.1.1 Out of Memory Errors

If the table is fragmented on many sstables on disk, you may run into trouble. Let me explain the reason. Your query is perfectly fine, but if you're querying a partition of, let's say 1 millions of rows spread across 10 SSTables, Cassandra may need to read the partition splits in all those SSTabl

conditional batches across two tables?

I’m trying to have some code acquire a lock by first at performing a table mutation, and then if it wins, performing a second table insert. I don’t think this is possible with batches though. I don’t think I can say “update this table, and if you are able to set the value, and the value doesn’t a

Re: Cassandra 2.1.1 Out of Memory Errors

DuyHai Doan, For wide rows, would it be better to switch to LeveledCompactionStrategy. The number of SSTables will decrease and it's also optimized for reading data. I have read in quite a few places that LeveledCompactionStrategy is better for wide rows. Is it true and would you recommend it?

Re: conditional batches across two tables?

Hi Kevin Have you looked at Cassandra's lightweight transactions? It sounds like they might work for you http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_ltwt_transaction_c.html. -- Daniel > On 17/11/2014, at 1:51 pm, Kevin Burton wrote: > > I’m trying to have some cod