Re: run cassandra on a small instance

Hey guys, After the upgrade to 2.1.3, and after almost exactly 5 hours running cassandra did indeed crash again on the 2GB ram VM. This is how the memory on the VM looked after the crash: [root@web2:~] #free -m total used free sharedbuffers cached Mem:

Re: Data tiered compaction and data model question

10-20 per minute is the average. Worstcase can be 10x of avg. On Wed, Feb 18, 2015 at 4:49 PM, Mohammed Guller wrote: > What is the maximum number of events that you expect in a day? What is > the worst-case scenario? > > > > Mohammed > > > > *From:* cass savy [mailto:casss...@gmail.com] > *Sen

Re: C* 2.1.2 invokes oom-killer

I neglected to mention, I also adjust the oom score of cassandra, to tell the kernel to kill something else other than cassandra. (Like if one of your dev’s runs a script that uses a lot of memory, so it kills your dev’s script instead). http://lwn.net/Articles/317814/

RE: run cassandra on a small instance

I asked this previously when a similar message came through, with a similar response. planetcassandra seems to have it “right”, in that stable=2.0, development=2.1, whereas the apache site says stable is 2.1. “Right” in they assume latest minor version is development. Why not have the apache s

Re: C* 2.1.2 invokes oom-killer

Are you tweaking the "nice" priority on Cassandra? (Type: man nice) if you don't know much about it. Certainly improving cassandra's nice score becomes important when you have other things running on the server like scheduled jobs of people logging in to the server and doing things. ___

Re: run cassandra on a small instance

Robert, Let me know if I’m off base about this—but I feel like I see a lot of posts that are like this (i.e., use this arbitrary version, not this other arbitrary version).  Why are releases going out if they’re “broken”?  This seems like a very confusing way for new (and existing) users to app

Re: run cassandra on a small instance

> > 2.1.2 is IMO broken and should not be used for any purpose. > Use 2.1.1 or 2.1.3. > https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ > =Rob Cool man. Thanks for the info. I just upgraded to 2.1.3. We'll see how that goes. I can let you know more once it's been runnin

Re: run cassandra on a small instance

On Wed, Feb 18, 2015 at 5:09 PM, Tim Dunphy wrote: > I'm attempting to run Cassandra 2.1.2 on a smallish 2.GB ram instance > over at Digital Ocean. It's a CentOS 7 host. > 2.1.2 is IMO broken and should not be used for any purpose. Use 2.1.1 or 2.1.3. https://engineering.eventbrite.com/what-ve

Logging client ID for YCSB workloads on Cassandra?

Hi, I'd like to log the client ID for every operation performed by the YCSB on my Cassandra cluster. The purpose is to identify & analyze various other consistency measures other than eventual consistency. I wanted to know if people have done something similar in the past. Or am I missing someth

run cassandra on a small instance

Hey all, I'm attempting to run Cassandra 2.1.2 on a smallish 2.GB ram instance over at Digital Ocean. It's a CentOS 7 host. But I'm having some difficulty there. Cassandra will start with no problems and run for a while. But then choke on the lack of memory and crash. This is what the system look

RE: Data tiered compaction and data model question

What is the maximum number of events that you expect in a day? What is the worst-case scenario? Mohammed From: cass savy [mailto:casss...@gmail.com] Sent: Wednesday, February 18, 2015 4:21 PM To: user@cassandra.apache.org Subject: Data tiered compaction and data model question We want to track

Data tiered compaction and data model question

We want to track events in log Cf/table and should be able to query for events that occurred in range of mins or hours for given day. Multiple events can occur in a given minute. Listed 2 table designs and leaning towards table 1 to avoid large wide row. Please advice on *Table 1*: not very wid

Re: Cassandra install on JRE vs JDK

...with JDK 1.7.x...(not 1.6.x but it's equal) Sorry... Regards. Karim Duran 2015-02-18 23:33 GMT+01:00 karim duran : > Hi Mark, Cass Savy, Robert... > > I confirm that Cassandra runs on JRE ( or JDK because a JRE is provided > with JDK ). > Oracle (ex Sun Microsystem) is the best choice to mak

Re: Cassandra install on JRE vs JDK

Hi Mark, Cass Savy, Robert... I confirm that Cassandra runs on JRE ( or JDK because a JRE is provided with JDK ). Oracle (ex Sun Microsystem) is the best choice to make Cassandra running without issue. (there is some problems with IBM JVM or OpenJDK). Here's a screenshot of Cassandra 2.1.2 runnin

Re: Cassandra install on JRE vs JDK

Cassandra 1.2.18 and Java 1.6 u45. Planning an upgrade to the 2.x series in the near future along with a bump in version of Java. Regards, Mark On 18 February 2015 at 20:32, cass savy wrote: > Thanks Mark for quick response. What version of Cassandra and JDK are you > using in Prod. > > > On

Re: Cassandra install on JRE vs JDK

Thanks Robert for quick response. I use Oracle JDK and not OpenJDK. On Wed, Feb 18, 2015 at 11:54 AM, Robert Stupp wrote: > The ”natural” dependency of Cassandra is the JRE (not the JDK) - e.g. in > the Debian package. > You should be safe using JRE instead of JDK. > > If you’re asking whether

Re: Cassandra install on JRE vs JDK

Thanks Mark for quick response. What version of Cassandra and JDK are you using in Prod. On Wed, Feb 18, 2015 at 11:58 AM, Mark Reddy wrote: > Yes you can use Oracle JDK if your prefer, I've been using the JDK with > Cassandra in production for years without issue. > > Regards, > Mark > > On 1

Re: Cassandra install on JRE vs JDK

Yes you can use Oracle JDK if your prefer, I've been using the JDK with Cassandra in production for years without issue. Regards, Mark On 18 February 2015 at 19:49, cass savy wrote: > Can we install Oracle JDK instead of JRE in Cassandra servers? We have > few clusters running JDK when we upgr

Re: Cassandra install on JRE vs JDK

The ”natural” dependency of Cassandra is the JRE (not the JDK) - e.g. in the Debian package. You should be safe using JRE instead of JDK. If you’re asking whether to use a non-Oracle JVM - the answer would be: use the Oracle JVM. OpenJDK might work, but I’d not recommend it. > Am 18.02.2015 um

Cassandra install on JRE vs JDK

Can we install Oracle JDK instead of JRE in Cassandra servers? We have few clusters running JDK when we upgraded to C*2.0. Is there any known issue or impact with using JDK vs JRE? What is the reason to not use Oracle JDK in C* servers? Is there any performance impact ? Please advice.

Re: Deleting Statistics.db at startup

On Wed, Feb 18, 2015 at 4:02 AM, Tomer Pearl wrote: > My question is what is the consequences of deleting this file every time > the node is starting up? Performance wise or other. > You waste the time Cassandra spends to regenerate it. I personally would not institute an operational practice

Re: C* 2.1.2 invokes oom-killer

On Wed, Feb 18, 2015 at 10:28 AM, Michał Łowicki wrote: > Couple of times a day 2 out of 4 members cluster nodes are killed > This sort of issue is usually best handled/debugged interactively on IRC. But briefly : - 2.1.2 is IMO broken for production. Downgrade (officially unsupported but fine

C* 2.1.2 invokes oom-killer

Hi, Couple of times a day 2 out of 4 members cluster nodes are killed root@db4:~# dmesg | grep -i oom [4811135.792657] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [6559049.307293] java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Nodes are us

Re: Many pending compactions

As Al Tobey suggest me I upgraded my 2.1.0 to snaphot version of 2.1.3. I have now installed exactly this build: https://cassci.datastax.com/job/cassandra-2.1/912/ I see many compaction which completes, but some of them are really slow. Maybe I should send some stats form OpsCenter or servers? But

Re: Many pending compactions

Ja, Please upgrade to official 2.1.3 we've fixed many things related to compaction. Are you seeing the compactions % complete progress at all? On Wed, Feb 18, 2015 at 11:58 AM, Roni Balthazar wrote: > Try repair -pr on all nodes. > > If after that you still have issues, you can try to rebuild t

Re: Many pending compactions

Try repair -pr on all nodes. If after that you still have issues, you can try to rebuild the SSTables using nodetool upgradesstables or scrub. Regards, Roni Balthazar > Em 18/02/2015, às 14:13, Ja Sam escreveu: > > ad 3) I did this already yesterday (setcompactionthrouput also). But still

Re: Many pending compactions

ad 3) I did this already yesterday (setcompactionthrouput also). But still SSTables are increasing. ad 1) What do you think I should use -pr or try to use incremental? On Wed, Feb 18, 2015 at 4:54 PM, Roni Balthazar wrote: > You are right... Repair makes the data consistent between nodes. >

Re: Many pending compactions

You are right... Repair makes the data consistent between nodes. I understand that you have 2 issues going on. You need to run repair periodically without errors and need to decrease the numbers of compactions pending. So I suggest: 1) Run repair -pr on all nodes. If you upgrade to the new 2.1.

Re: Many pending compactions

Cassandra 2.1 comes with incremental repair, and I haven't read the details myself: http://www.datastax.com/documentation/cassandra/2.1/cassandra/operations/ops_repair_nodes_c.html However, AFAIK, a full repair will rebuild all sstables, that's why you should have more than 50% of disk space av

Re: Many pending compactions

Can you explain me what is the correlation between growing SSTables and repair? I was sure, until your mail, that repair is only to make data consistent between nodes. Regards On Wed, Feb 18, 2015 at 4:20 PM, Roni Balthazar wrote: > Which error are you getting when running repairs? > You need

Re: Many pending compactions

Which error are you getting when running repairs? You need to run repair on your nodes within gc_grace_seconds (eg: weekly). They have data that are not read frequently. You can run "repair -pr" on all nodes. Since you do not have deletes, you will not have trouble with that. If you have deletes, i

Re: Many pending compactions

1) we tried to run repairs but they usually does not succeed. But we had Leveled compaction before. Last week we ALTER tables to STCS, because guys from DataStax suggest us that we should not use Leveled and alter tables in STCS, because we don't have SSD. After this change we did not run any repai

Re: Many pending compactions

Are you running repairs within gc_grace_seconds? (default is 10 days) http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_repair_nodes_c.html Double check if you set cold_reads_to_omit to 0.0 on tables with STCS that you do not read often. Are you using default values for

Re: Adding new node to cluster

Hello, Please note that DataStax has updated the documentation for replacing a seed node. The new docs outline a simplified process to help avoid the confusion on this topic. http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_seed_node.html Jonathan [image:

Re: Many pending compactions

I don't have problems with DC_B (replica) only in DC_A(my system write only to it) I have read timeouts. I checked in OpsCenter SSTable count and I have: 1) in DC_A same +-10% for last week, a small increase for last 24h (it is more than 15000-2 SSTables depends on node) 2) in DC_B last 24h

Re: Many pending compactions

Hi, You can check if the number of SSTables is decreasing. Look for the "SSTable count" information of your tables using "nodetool cfstats". The compaction history can be viewed using "nodetool compactionhistory". About the timeouts, check this out: http://www.datastax.com/dev/blog/how-cassandra-

Deleting Statistics.db at startup

Hello, I have received the following error ERROR [SSTableBatchOpen:2] 2015-01-19 13:55:28,478 CassandraDaemon.java (line 196) Exception in thread Thread[SSTableBatchOpen:2,5,main] java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistog

Re: Many pending compactions

Hi, Thanks for your "tip" it looks that something changed - I still don't know if it is ok. My nodes started to do more compaction, but it looks that some compactions are really slow. In IO we have idle, CPU is quite ok (30%-40%). We set compactionthrouput to 999, but I do not see difference. Can

Re: Adding new node to cluster

Hello, I have decommissioned a node, deleted data,commitlog and saved caches, changed yaml file to not include self ip and started it. For some reason I do not fully understand, Opscenter says that the node is in an unknown datacenter. Nodetool says UJ but shows "?" in the Owns column. I have st