Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-07 Thread Shravan C
OOM. Thanks everyone of you. From: Jeff Jirsa Sent: Tuesday, March 7, 2017 1:19 PM To: user@cassandra.apache.org Subject: Re: OOM on Apache Cassandra on 30 Plus node at the same time On 2017-03-03 09:18 (-0800), Shravan Ch wrote: > > nodetool compactio

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-07 Thread Jeff Jirsa
On 2017-03-03 09:18 (-0800), Shravan Ch wrote: > > nodetool compactionstats -H > pending tasks: 3 > compaction typekeyspace table > completed totalunit progress > Compaction system hints > 28.

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-07 Thread Jeff Jirsa
On 2017-03-04 07:23 (-0800), "Thakrar, Jayesh" wrote: > LCS does not rule out frequent updates - it just says that there will be more > frequent compaction, which can potentially increase compaction activity > (which again can be throttled as needed). > But STCS will guarantee OOM when you h

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-06 Thread Eric Evans
On Fri, Mar 3, 2017 at 11:18 AM, Shravan Ch wrote: > More than 30 plus Cassandra servers in the primary DC went down OOM > exception below. What puzzles me is the scale at which it happened (at the > same minute). I will share some more details below. You'd be surprised; When it's the result of a

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-04 Thread Thakrar, Jayesh
s , "user@cassandra.apache.org" Subject: Re: OOM on Apache Cassandra on 30 Plus node at the same time I was looking at nodetool info across all nodes. Consistently JVM heap used is ~ 12GB and off heap is ~ 4-5GB. From: Thakrar, Jayesh Sent: Saturday, M

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-04 Thread Priyanka
Sent from my iPhone > On Mar 3, 2017, at 12:18 PM, Shravan Ch wrote: > > Hello, > > More than 30 plus Cassandra servers in the primary DC went down OOM exception > below. What puzzles me is the scale at which it happened (at the same > minute). I will share some more details below. > > Sy

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-04 Thread Shravan C
have a look at the offheap + onheap size of our jvm using "nodetool -info" ? From: Shravan C Date: Friday, March 3, 2017 at 11:11 PM To: Joaquin Casares , "user@cassandra.apache.org" Subject: Re: OOM on Apache Cassandra on 30 Plus node at the same time We run C* at 3

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-04 Thread Edward Capriolo
> > > Thanks, > > Shravan > -- > > *From:* Thakrar, Jayesh > > *Sent:* Friday, March 3, 2017 3:47:27 PM > *To:* Joaquin Casares; user@cassandra.apache.org > > *Subject:* Re: OOM on Apache Cassandra on 30 Plus node at the same time

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-04 Thread Thakrar, Jayesh
size of our jvm using "nodetool -info" ? From: Shravan C Date: Friday, March 3, 2017 at 11:11 PM To: Joaquin Casares , "user@cassandra.apache.org" Subject: Re: OOM on Apache Cassandra on 30 Plus node at the same time We run C* at 32 GB and all servers have 96GB RAM. We use

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-03 Thread Shravan C
docs.scylladb.com Scylla is a Cassandra-compatible NoSQL data store that can handle 1 million transactions per second on a single server. From: Joaquin Casares Date: Friday, March 3, 2017 at 11:34 AM To: Subject: Re: OOM on Apache Cassandra on 30 Plus node at the same time Hello Shravan,

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-03 Thread Thakrar, Jayesh
better and should be the default (my opinion) unless you want DTCS A good description of all three compactions is here - http://docs.scylladb.com/kb/compaction/ From: Joaquin Casares Date: Friday, March 3, 2017 at 11:34 AM To: Subject: Re: OOM on Apache Cassandra on 30 Plus node at the same

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-03 Thread Shravan C
From: Joaquin Casares Sent: Friday, March 3, 2017 11:34:58 AM To: user@cassandra.apache.org Subject: Re: OOM on Apache Cassandra on 30 Plus node at the same time Hello Shravan, Typically asynchronous requests are recommended over batch statements since batch statements will cause more work on

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-03 Thread Joaquin Casares
Hello Shravan, Typically asynchronous requests are recommended over batch statements since batch statements will cause more work on the coordinator node while individual requests, when using a TokenAwarePolicy, will hit a specific coordinator, perform a local disk seek, and return the requested in

OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-03 Thread Shravan Ch
Hello, More than 30 plus Cassandra servers in the primary DC went down OOM exception below. What puzzles me is the scale at which it happened (at the same minute). I will share some more details below. System Log: http://pastebin.com/iPeYrWVR GC Log: http://pastebin.com/CzNNGs0r