Tuning Cassandra for event sourcing model

Hi all, We use Cassandra as a primary operational database in our data platform. One of the cases - store the latest user profiles information that is sourced from the upstream system. Importantly, we get each time the entire row, not particular columns that were updated. The access pattern is th

Re: Tuning Cassandra 2.1 for High Writes and Immediate Reads

If you use clustering order by and need to keep the top rows in cache, look at the row cache in 2.1. http://www.datastax.com/dev/blog/row-caching-in-cassandra-2-1 On Jun 16, 2015 5:39 AM, "Pracheer Agarwal" wrote: > Hi, > > We are evaluating Cassandra 2.1 for our new production system. The > fol

Tuning Cassandra 2.1 for High Writes and Immediate Reads

Hi, We are evaluating Cassandra 2.1 for our new production system. The following are the requirements: 1. 15K writes/sec with 5 KB blob in a single column of a column family, 2. This is followed by immediate Reads by multiple consumer threads, the read requires us to return entire Row and not onl

Re: Tuning cassandra (compactions overall)

I already had this kind of trouble while repairing a month ago. I have problems that I am the only one to have. I guess I have something wrong either in the configuration of my nodes or in my data that makes them wrong after a restart/repair. I am planning to try deploying an EC2 cluster with data

Re: Tuning cassandra (compactions overall)

I've not heard of anything like that in the recent versions. There were some issues in the early 0.8 https://github.com/apache/cassandra/blob/trunk/NEWS.txt#L383 If you are on a recent version can you please create a jira ticket https://issues.apache.org/jira/browse/CASSANDRA describing what yo

Re: Tuning cassandra (compactions overall)

"not sure what you mean by And after restarting the second one I have lost all the consistency of my data. All my statistics since September are totally false now in production Can you give some examples?" After restarting my 2 nodes (one after the other), All my counters have become wrong. The c

Re: Tuning cassandra (compactions overall)

not sure what you mean by > And after restarting the second one I have lost all the consistency of > my data. All my statistics since September are totally false now in > production Can you give some examples? Counter are not idempotent so if the client app retries TimedOut requests you can get

Re: Tuning cassandra (compactions overall)

Hi Aaron. I wanted to try the new config. After doing a rolling restart I have all my counters false, with wrong values. I stopped my servers with the following : nodetool -h localhost disablegossip nodetool -h localhost disablethrift nodetool -h localhost drain kill cassandra sigterm (15) via ht

Re: Tuning cassandra (compactions overall)

> What is the the benefit of having more memory ? I mean, I don't > understand why having 1, 2, 4, 8 or 16 GB of memory is so different. Less frequent and less aggressive garbage collection frees up CPU resources to run the database. Less memory results in frequent and aggressive (i.e. stop the

Re: Tuning cassandra (compactions overall)

Using c1.medium, we are currently able to deliver the service. What is the the benefit of having more memory ? I mean, I don't understand why having 1, 2, 4, 8 or 16 GB of memory is so different. In my mind, Cassandra will fill the heap and from then, start to flush and compact to avoid OOMing and

Re: Tuning cassandra (compactions overall)

> 1 - I got this kind of message quite often (let's say every 30 seconds) : You are running out of memory. Depending on the size of your schema and the work load you will want to start with 4 or 8 GB machines. But most people get the best results with 16Gb. On AWS the common setup is to use m1.x

Tuning cassandra (compactions overall)

Hi, I'm using a 2 node cluster in production ( 2 EC2 c1.medium, CL.ONE, RF = 2, using RP) 1 - I got this kind of message quite often (let's say every 30 seconds) : WARN [ScheduledTasks:1] 2012-05-15 15:44:53,083 GCInspector.java (line 145) Heap is 0.8081418550931491 full. You may need to reduce

Any suggestions tuning Cassandra

Hello Folks, We are evaluating cassandra for one of our storage needs. I am running a benchmark test to gauge cassandra's performance using http://github.com/brianfrankcooper/YCSB/wiki Setup for Cassandra is 5 node cluster, replication factor 3. CentOS55 on amazon ec2 Sample test data single field

Re: Tuning cassandra to use less memory

+1 on disabling swap On Oct 7, 2010 3:27 PM, "Peter Schuller" wrote: >> The nodes are still swapping, even though the swappiness is set to zero >> right now. After swapping comes the OOM. > > In addition to what's already been said, consider just flat out > disabling swap completely, unless you ha

Re: Tuning cassandra to use less memory

> The nodes are still swapping, even though the swappiness is set to zero > right now. After swapping comes the OOM. In addition to what's already been said, consider just flat out disabling swap completely, unless you have other things on the machine that cause swap to be significantly useful (i.

Re: Tuning cassandra to use less memory

There is an explanation of how to lock the JVM into memory here http://www.riptano.com/blog/whats-new-cassandra-065However from the JVM Heap Size section here http://wiki.apache.org/cassandra/MemtableThresholdsFor a rough rule of thumb, Cassandra's internal datastructures will require about memtabl

Re: Tuning cassandra to use less memory

On 10/6/10 9:05 AM, Utku Can Topçu wrote: The nodes are still swapping, even though the swappiness is set to zero right now. After swapping comes the OOM. https://issues.apache.org/jira/browse/CASSANDRA-1214 ? =Rob

Re: Tuning cassandra to use less memory

Hi Oleg, I've been also looking into these after some research. I've been tacking with: 1. Setting the default max and min heap from 1G to 1500M. 2. I'm not using row caches, and the key caches are set to 1000, before they were 200K as default 3. I've lowered the memtable throughput to 32MB 4. We

Re: Tuning cassandra to use less memory

> > Hi All,We're currently starting to get OOM exceptions in our cluster. I'm trying to push the limiations of our machines. Currently we have 1.7 G memory (ec2-medium)I'm wondering if by tweaking some of cassandra's configuration settings, is it possible to make it live in peace and less memory.

Tuning cassandra to use less memory

Hi All, We're currently starting to get OOM exceptions in our cluster. I'm trying to push the limiations of our machines. Currently we have 1.7 G memory (ec2-medium) I'm wondering if by tweaking some of cassandra's configuration settings, is it possible to make it live in peace and less memory :)

Re: Tuning Cassandra

Using multiple client threads (w/ pooled thrift connections) will be even better than mutating really large chunks at a time. On Tue, May 11, 2010 at 4:16 AM, David Boxenhorn wrote: > Turns out the problem is with batch mutate. I mutate chunks 100 times > bigger, it goes 100 times faster. > > Now

Re: Tuning Cassandra

Turns out the problem is with batch mutate. I mutate chunks 100 times bigger, it goes 100 times faster. Now I have a problem with running out of memory sometimes On Mon, May 10, 2010 at 8:17 PM, B. Todd Burruss wrote: > have you put your commit log on a disk by itself? not a logical parti

Re: Tuning Cassandra

The performance you are describing is completely abnormal. The first step in troubleshooting it is profiling your client behavior because that is almost certainly where the problem is. Where is it spending its time? If that ultimately indicates it is really waiting on Cassandra, you can turn you

Re: Tuning Cassandra

have you put your commit log on a disk by itself? not a logical partition shared by oracle or cassandra "data". this will make a difference, as you don't want the cassandra commit logs competing with other OS and oracle I/O. look in storage-conf.xml and see if you can move this. also check

Re: Tuning Cassandra

David, Are you using batchMutate or insert? Jump on over to hector-users if you want API help with either of these. -Nate

Re: Tuning Cassandra

TBufferedTransport is a C# thing. It's not necessary in Java. On Mon, May 10, 2010 at 3:48 AM, Ran Tavory wrote: > Hector uses tsocket. not sure what you mean by "buffered" - is that framed? > Hector by default does not use framed. > The code is here if you'd like to have a > look http://github.

Re: Tuning Cassandra

That sounds like "Did you start beating your wife recently or have you been beating her for a long time now?" :) Ron On Mon, 2010-05-10 at 09:00 -0700, Paul Prescod wrote: > Does the Caasandra performance start fast and slow down (indicating > some buffer being filled) or does it start slow and

Re: Tuning Cassandra

Does the Caasandra performance start fast and slow down (indicating some buffer being filled) or does it start slow and stay slow? On Mon, May 10, 2010 at 2:05 AM, David Boxenhorn wrote: > I read something like 80,000 rows from Oracle and write them to Cassandra in > chunks of 1000 rows - so I'm

Re: Tuning Cassandra

http://mail-archives.apache.org/mod_mbox/cassandra-user/201005.mbox/%3cd0c18662921df14f983c53625de8a7241e4c11f...@34093-mbx-c14.mex07a.mlsrvr.com%3e > > > > > > From: Ran Tavory [mailto:ran...@gmail.com] > Sent: May 10, 2010 11:48 AM > To: user@cassandra.apache.org > Subject: R

Re: Tuning Cassandra

I read something like 80,000 rows from Oracle and write them to Cassandra in chunks of 1000 rows - so I'm supposedly working to Cassandra's strength and Oracle's weakness. Reading 1000 rows from Oracle is "instantaneous", writing them takes maybe 30 seconds. Not too much data per row, maybe 1K.

RE: Tuning Cassandra

: user@cassandra.apache.org Subject: Re: Tuning Cassandra Hector uses tsocket. not sure what you mean by "buffered" - is that framed? Hector by default does not use framed. The code is here if you'd like to have a look http://github.com/rantav/hector/blob/master/src/main/java

Re: Tuning Cassandra

Hector uses tsocket. not sure what you mean by "buffered" - is that framed? Hector by default does not use framed. The code is here if you'd like to have a look http://github.com/rantav/hector/blob/master/src/main/java/me/prettyprint/cassandra/service/CassandraClientFactory.java#L77

Re: Tuning Cassandra

You asked for it! You might want to skip to Cassandra.save() ... public class BuildCassandraDB { private static void importContentInterests(Connection connection) throws Exception { Statement statement; ResultSet resultSet; String sql; int count, totalcount;

Re: Tuning Cassandra

What is the complete code string you are using to connect with cassandra from Java code On Mon, May 10, 2010 at 1:49 PM, David Boxenhorn wrote: > I don't know what "TSocket or the buffered one" means. Maybe I should know? > > I'm using Hector. Does that explain anything? > > On Mon, May 10, 20

Re: Tuning Cassandra

I don't know what "TSocket or the buffered one" means. Maybe I should know? I'm using Hector. Does that explain anything? On Mon, May 10, 2010 at 11:15 AM, vd wrote: > > Hi > > what is it that you are using to connect with cassnadra TSocket or the > buffered one ? > > >

Re: Tuning Cassandra

Hi what is it that you are using to connect with cassnadra TSocket or the buffered one ? ___ On Mon, May 10, 2010 at 1:29 PM, David Boxenhorn wrote: > I'm running Java on the client, jdbc queries on Oracle, Hector on

Re: Tuning Cassandra

I'm running Java on the client, jdbc queries on Oracle, Hector on Cassandra. The Cassandra and Oracle database designs are radically different, as you might guess. I have no doubt that Cassandra can be tuned, in a multiple-server cluster, to have superior throughput (that's why I'm doing it!). Bu

Re: Tuning Cassandra

Hi David If I may ask...how do you plan to import data from oracle to cassandra ? As answer AFAIK cassandra's true ability comes into play when running on more than one machine...and please share how you are making comparisons like on writes or reads from cassandra.

Tuning Cassandra

I'm running Oracle and Cassandra on my machine, trying to import my data to Cassandra from Oracle. In my configuration Oracle is about ten times faster than Cassandra. Cassandra has out-of-the-box tuning. I am new to Cassandra. How do I begin trying to tune it? Thanks.