Re: High GC activity on node with 4TB on data

Sorry to jump on this late. GC is one of my favorite topics. A while ago I wrote a blob post about C* GC tuning and documented several issues that I had experienced. It seems it has helped some people in that past, so I am sharing it here: http://aryanet.com/blog/cassandra-garbage-collector-tuning

Re: C* 2.1.2 invokes oom-killer

We don't have other things running on these boxes and C* is consuming all the memory. Will try to upgrade to 2.1.3 and if won't help downgrade to 2.1.2.  — Michał On Thu, Feb 19, 2015 at 2:39 AM, Jacob Rhoden wrote: > Are you tweaking the "nice" priority on Cassandra? (Type: man nice) if

Re: run cassandra on a small instance

I have Cassandra instances running on VMs with smaller RAM (1GB even) and I don't go OOM when testing them. Although I use them in AWS and other providers, never tried Digital Ocean. Does Cassandra just fails after some time running or it is failing on some specific read/write? Regards, Carlos J

Re: C* 2.1.2 invokes oom-killer

Can you check how many SSTables you have? It is more or less a know fact that 2.1.2 has lots of problems with compaction so a upgrade can solve it. But a high number of SSTables can confirm that indeed compaction is your problem not something else. Regards, Carlos Juzarte Rolo Cassandra Consultan

Re: run cassandra on a small instance

What does your schema look like, your total data size and your read/write patterns? Maybe you are simply doing a heavier workload than a small instance can handle. Regards, Mark On 19 February 2015 at 08:40, Carlos Rolo wrote: > I have Cassandra instances running on VMs with smaller RAM (1GB e

Cancel subscription

Cancel subscription

Re: C* 2.1.2 invokes oom-killer

In all tables SSTable counts is below 30. On Thu, Feb 19, 2015 at 9:43 AM, Carlos Rolo wrote: > Can you check how many SSTables you have? It is more or less a know fact > that 2.1.2 has lots of problems with compaction so a upgrade can solve it. > But a high number of SSTables can confirm that i

Re: Many pending compactions

Hi, 2.1.3 is now the official latest release - I checked this morning and got this good surprise. Now it's update time - thanks to all guys involved, if I meet anyone one beer from me :-) The changelist is rather long: https://git1-us-west.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=C

Re: C* 2.1.2 invokes oom-killer

So compaction doesn't seem to be your problem (You can check with nodetool compactionstats just to be sure). How much is your write latency on your column families? I had OOM related to this before, and there was a tipping point around 70ms. -- --

Re: C* 2.1.2 invokes oom-killer

On Thu, Feb 19, 2015 at 10:41 AM, Carlos Rolo wrote: > So compaction doesn't seem to be your problem (You can check with nodetool > compactionstats just to be sure). > pending tasks: 0 > > How much is your write latency on your column families? I had OOM related > to this before, and there was

Re: C* 2.1.2 invokes oom-killer

Do you have trickle_fsync enabled? Try to enable that and see if it solves your problem, since you are getting out of non-heap memory. Another question, is always the same nodes that die? Or is 2 out of 4 that die? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo

Re: C* 2.1.2 invokes oom-killer

On Thu, Feb 19, 2015 at 11:02 AM, Carlos Rolo wrote: > Do you have trickle_fsync enabled? Try to enable that and see if it solves > your problem, since you are getting out of non-heap memory. > > Another question, is always the same nodes that die? Or is 2 out of 4 that > die? > Always the same

Re: C* 2.1.2 invokes oom-killer

|trickle_fsync| has been enabled for long time in our settings (just noticed): trickle_fsync: true trickle_fsync_interval_in_kb: 10240 On Thu, Feb 19, 2015 at 12:12 PM, Michał Łowicki wrote: > > > On Thu, Feb 19, 2015 at 11:02 AM, Carlos Rolo wrote: > >> Do you have trickle_fsync enabled? Try

Re: Cancel subscription

Please use user-unsubscr...@cassandra.apache.org to unsubscribe from this mailing list. Thanks Regards, Mark On 19 February 2015 at 09:14, Hilary Albutt - CEO < hil...@incrediblesoftwaresolutions.com> wrote: > Cancel subscription >

can't delete tmp file

Hi guys: I have a 20 nodes C* cluster with vnodes,version is 2.1.2. when after compaction some nodes has some tmp files (like this: weibo-content1-tmp-ka-95117-Data.db),there can't remove after GC. I want know how to deal this,and It's normal? Thanks. -- 曹志富 手机

Node joining take a long time

Hi guys: I have a 20 nodes C* cluster with vnodes,version is 2.1.2. When I add a node to my cluster,it take a long time ,and somes exists node nodetool nestats show this: Mode: NORMAL Unbootstrap cfe03590-b02a-11e4-95c5-b5f6ad9c7711 /172.19.105.49 Receiving 68 files, 23309801005 bytes

Re: can't delete tmp file

Hi, try 2.1.3 - with 2.1.2 this is "normal". From the changelog: * Make sure we don't add tmplink files to the compaction strategy (CASSANDRA-8580) * Remove tmplink files for offline compactions (CASSANDRA-8321) In most cases they are safe to delete, I did this when the node was down. Cheers

Re: can't delete tmp file

Just upgrade my cluster to 2.1.3??? -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-02-19 20:32 GMT+08:00 Roland Etzenhammer : > Hi, > > try 2.1.3 - with 2.1.2 this is "normal". From the changelog: > > * Make sure we don't add t

Re: can't delete tmp file

Thanks you Roland -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-02-19 20:32 GMT+08:00 Roland Etzenhammer : > Hi, > > try 2.1.3 - with 2.1.2 this is "normal". From the changelog: > > * Make sure we don't add tmplink files to th

Re: can't delete tmp file

You should upgrade to 2.1.3 for sure. Check the changelog here: https://git1-us-west.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-2.1.3 Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedi

Re: C* 2.1.2 invokes oom-killer

Then you are probably hitting a bug... Trying to find out in Jira. The bad news is the fix is only to be released on 2.1.4. Once I find it out I will post it here. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com

Re: Data tiered compaction and data model question

What's the typical size of the data field? Unless it's very large, I don't think table 2 is a "very" wide row (10x20x60x24=288000 events/partition at worst). Plus you only need to store 30 days of data. The over data size is 288000x30=8,640,000 events. I am not even sure if you need C* depending on

Re: Node joining take a long time

What is a long time in your scenario? What is the data size in your cluster? I'm sure Rob will be along shortly to say that 2.1.2 is, in his opinion, broken for production use...an opinion I'd agree with. So bare that in mind if you are running a production cluster. Regards, Mark On 19 February

Re: run cassandra on a small instance

> > I have Cassandra instances running on VMs with smaller RAM (1GB even) and > I don't go OOM when testing them. Although I use them in AWS and other > providers, never tried Digital Ocean. > Does Cassandra just fails after some time running or it is failing on some > specific read/write? Hi Ca

Re: run cassandra on a small instance

What I normally do is install plain CentOS (Not any AMI build for Cassandra) and I don't use them for production! I run them for testing, fire drills and some cassandra-stress benchmarks. I will look if I had more than 5h Cassandra uptime. I can even put one up now and do the test and get the resul

Re: run cassandra on a small instance

> > What I normally do is install plain CentOS (Not any AMI build for > Cassandra) and I don't use them for production! I run them for testing, > fire drills and some cassandra-stress benchmarks. I will look if I had more > than 5h Cassandra uptime. I can even put one up now and do the test and get

Re: run cassandra on a small instance

> > What does your schema look like, your total data size and your read/write > patterns? Maybe you are simply doing a heavier workload than a small > instance can handle. Hi Mark, OK well as mentioned this is all test data with almost literally no workload. So I doubt it's the data and/ or wor

Re: run cassandra on a small instance

On Wed, Feb 18, 2015 at 5:26 PM, Andrew wrote: > Let me know if I’m off base about this—but I feel like I see a lot of > posts that are like this (i.e., use this arbitrary version, not this other > arbitrary version). Why are releases going out if they’re “broken”? This > seems like a very conf

Re: Node joining take a long time

On Thu, Feb 19, 2015 at 7:34 AM, Mark Reddy wrote: > I'm sure Rob will be along shortly to say that 2.1.2 is, in his opinion, > broken for production use...an opinion I'd agree with. So bare that in mind > if you are running a production cluster. > If you speak of the devil, he will appear. But

Re: run cassandra on a small instance

One welcome change is http://cassandra.apache.org/ actually starts displaying: "Latest release *2.1.3* (Changes ), Stable release *2.0.12* (Changes

unsubscribe

Unsubscribe What is it going to take to get my address off this list? You're instructions are not working for me.

Re: unsubscribe

I don't believe that this email was meant for me... 2015-02-19 21:26 GMT+01:00 Redmumba : > Just send an email to user-unsubscr...@cassandra.apache.org > > It will respond back if there were any issues. > > On Thu, Feb 19, 2015 at 12:06 PM, Steve Ryder > wrote: > >> Unsubscribe >> >> >> >> What

Re: unsubscribe

Just send an email to user-unsubscr...@cassandra.apache.org It will respond back if there were any issues. On Thu, Feb 19, 2015 at 12:06 PM, Steve Ryder wrote: > Unsubscribe > > > > What is it going to take to get my address off this list? You’re > instructions are not working for me. > > > >

Re: Data tiered compaction and data model question

Any feedback on data tiered compaction? Have anybody used it? On Thu, Feb 19, 2015 at 6:06 AM, Kai Wang wrote: > What's the typical size of the data field? Unless it's very large, I don't > think table 2 is a "very" wide row (10x20x60x24=288000 events/partition at > worst). Plus you only need to

Re: C* 2.1.2 invokes oom-killer

Upgrade to 2.1.3 seems to help so far. After ~12 hours total memory consumption grew from 10GB to 10.5GB. On Thu, Feb 19, 2015 at 2:02 PM, Carlos Rolo wrote: > Then you are probably hitting a bug... Trying to find out in Jira. The bad > news is the fix is only to be released on 2.1.4. Once I fin

RE: Data tiered compaction and data model question

Reading 288,000 rows from a partition may cause problems. It is recommended not to read more than 100k rows in a partition ((although paging may help). So Table 2 may cause issues. I agree with Kai that for you may not even need C* for this use-case. C* is ideal for data with 3 Vs: volume, vel

[ANNOUNCE] Apache Gora 0.6 Released

Hi Folks, The Apache Gora team are pleased to announce the immediate availability of Apache Gora 0.6. This release addresses a modest 47 issues with some being major improvements, new functionality and dependency upgrades. Most notably the release involves key upgra

Re: Node joining take a long time

First thank all of you. Almost three days,till right now the status is still Joining. My cluster per 650G a node. -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-02-20 3:16 GMT+08:00 Robert Coli : > On Thu, Feb 19, 2015 at 7:34

Re: [ANNOUNCE] Apache Gora 0.6 Released

Congras! On Feb 20, 2015 2:59 AM, "Lewis John Mcgibbney" wrote: > Hi Folks, > > The Apache Gora team are pleased to announce the immediate availability of > Apache Gora 0.6. > > This release addresses a modest 47 issues > with some being major improvements, new func

Why no virtual nodes for Cassandra on EC2?

Hi all, The guide for installing Cassandra on EC2 says that "Note: The DataStax AMI does not install DataStax Enterprise nodes with virtual nodes enabled." http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installAMI.html Just curious why this is the case

Re: Node joining take a long time

So ,what can I do???Waiting for 2.1.4 or upgrade to 2.1.3?? -- 曹志富 手机：18611121927 邮箱：caozf.zh...@gmail.com 微博：http://weibo.com/boliza/ 2015-02-20 3:16 GMT+08:00 Robert Coli : > On Thu, Feb 19, 2015 at 7:34 AM, Mark Reddy > wrote: > >> I'm sure Rob will be alo

designing table

I am trying to design a table in Cassandra in which I will have multiple JSON String for a particular client id. abc123 - jsonA abc123 - jsonB abcd12345 - jsonC My query pattern is going to be - Give me all JSON String for a particular client id. Gi

Re: Data tiered compaction and data model question

Hi Cass, just a hint from the off - if I got it right you have: Table 1: PRIMARY KEY ( (event_day,event_hr),event_time) Table 2: PRIMARY KEY (event_day,event_time) Assuming your events to write come in by wall clock time, the first table design will have a hotspot on a specific node getting al