Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-03 Thread Shravan C
We run C* at 32 GB and all servers have 96GB RAM. We use STCS . LCS is not an option for us as we have frequent updates. Thanks, Shravan From: Thakrar, Jayesh Sent: Friday, March 3, 2017 3:47:27 PM To: Joaquin Casares; user@cassandra.apache.org Subject: Re: OOM

Re: Question on configuring Apache Cassandra with LDAP

2017-03-03 Thread Daniel Kleviansky
That is in fact my linked WIP branch, and I can confirm progress was stalled just to other priorities being set! It got to the point (in draft form) where authentication was working, but still need to work on extending IRoleManager. I will say this, while I initially started using the Apache LDA

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-03 Thread Thakrar, Jayesh
Had been fighting a similar battle, but am now over the hump for most part. Get info on the server config (e.g. memory, cpu, free memory (free -g), etc) Run "nodetool info" on the nodes to get heap and off-heap sizes Run "nodetool tablestats" or "nodetool tablestats ." on the key large tables Ess

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-03 Thread Shravan C
Hi Joaquin, We have inserts going into a tracking table. Tracking table is a simple table [PRIMARY KEY (comid, status_timestamp) ] with a few tracking attributes and sorted by status_timestamp. From a volume perspective it is not a whole lot. Thanks, Shravan F

Secondary indexes don't return expected results in 2.2.3

2017-03-03 Thread Enrique Bautista
Hello everyone, I run a 2.2.3 cluster and I've found that sometimes queries by secondary indexes don't return the expected results, even though the data is in the table. I'm sure the data is in there because queries by partition key or without WHERE clause do return the expected data. The only is

Re: Question on configuring Apache Cassandra with LDAP

2017-03-03 Thread Sam Tunnicliffe
This is something that has been discussed before and there's an JIRA open for it already, it looks like progress has stalled but you might get some pointers from the linked WIP branch. https://issues.apache.org/jira/browse/CASSANDRA-12294 On Fri, Mar 3, 2017 at 7:45 PM, Harika Vangapelli -T (hva

Question on configuring Apache Cassandra with LDAP

2017-03-03 Thread Harika Vangapelli -T (hvangape - AKRAYA INC at Cisco)
I am trying to configure Cassnadra with LDAP , and I am trying to write code and want to extend the functionality of org.apache.cassandra.auth.PasswordAuthenticator and override authenticate method but As PlainTextSaslAuthenticator innerclass has a private scope not able to use method overridin

Re: question of keyspace that just disappeared

2017-03-03 Thread George Webster
I think it does on drop keyspace. We had a recent enough snapshot so it wasn't a big deal to recover. However, we didn't have a snapshot for when the keyspace disappeared. @Romain: I believe you are correct about reliability. We just had a repair --full fail and CPU lock up one of the nodes at 100

Re: OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-03 Thread Joaquin Casares
Hello Shravan, Typically asynchronous requests are recommended over batch statements since batch statements will cause more work on the coordinator node while individual requests, when using a TokenAwarePolicy, will hit a specific coordinator, perform a local disk seek, and return the requested in

OOM on Apache Cassandra on 30 Plus node at the same time

2017-03-03 Thread Shravan Ch
Hello, More than 30 plus Cassandra servers in the primary DC went down OOM exception below. What puzzles me is the scale at which it happened (at the same minute). I will share some more details below. System Log: http://pastebin.com/iPeYrWVR GC Log: http://pastebin.com/CzNNGs0r

New command line client for cassandra-reaper

2017-03-03 Thread Vincent Rischmann
Hi, I'm using cassandra-reaper (https://github.com/thelastpickle/cassandra-reaper) to manage repairs of my Cassandra clusters, probably like a bunch of other people. When I started using it (it was still the version from the spotify repository) the UI didn't work well, and the Python cli clien

Re: question of keyspace that just disappeared

2017-03-03 Thread Edward Capriolo
On Fri, Mar 3, 2017 at 7:56 AM, Romain Hardouin wrote: > I suspect a lack of 3.x reliability. Cassandra could had gave up with > dropped messages but not with a "drop keyspace". I mean I already saw some > spark jobs with too much executors that produce a high load average on a > DC. I saw a C* n

Re: question of keyspace that just disappeared

2017-03-03 Thread Romain Hardouin
I suspect a lack of 3.x reliability. Cassandra could had gave up with dropped messages but not with a "drop keyspace". I mean I already saw some spark jobs with too much executors that produce a high load average on a DC. I saw a C* node with a 1 min. load avg of 140 that can still have a P99 re

Re: question of keyspace that just disappeared

2017-03-03 Thread George Webster
Thank you for your reply and good to know about the debug statement. I haven't We never dropped or re-created the keyspace before. We haven't even performed writes to that keyspace in months. I also checked the permissions of Apache, that user had read only access. Unfortunately, I reverted from

Re: Attached profiled data but need help understanding it

2017-03-03 Thread Romain Hardouin
Also, I should have mentioned that it would be a good idea to spawn your three benchmark instances in the same AZ, then try with one instance on each AZ to see how network latency affects your LWT rate. The lower latency is achievable with three instances on the same placement group of course bu