It seems it was linked to "data volume". The symptoms were : - a read query is executed on a server (#3 in our case). this query is like select from CF where A= and B= and C= and D= where A..D are secondary indexes. - #3 CPU increased a lot (CPU load around 20 for 8 cores). During that time, cluster is still under load - then a couple of seconds later, server #2 and #4 (next and previous tokens...) are impacted (CPU load around 20 for 8 cores).
- from the client side, we got HTTP time out. - ReadStage pending was high. - no IO (disk, network), just CPU bursts - We stopped the load and after a while (around 10 minute), the cluster calm down. Indeed the query we executed was reading a lot of data (too much ;) ) What we did (this morning in fact) : - we precalculated the indexes we need in order to be sure to always have a query like select fom CF where idx= (and use only one index) - so our data model is composed of columns and "concatened" indexes. these "concatened indexes" are secondary indexes (into cassandra). In our case and based on our data access, we have 4 precalculated indexes. It works terribly well (thanks to my colleague Franck for having this idea). Yet, I would be strongly interested in internals about secondary indexes. What I understood is - secondary index is local to the seed and index local data only. - concerning the fault-tolerance, if you have a RF to 3 and lost 2 consecutive seeds / token, you are dead for all the reads you are doing (even if the down seeds do not have the data but the query need to read the index on the down seeds). Am I right? On Wed, Jul 4, 2012 at 11:13 AM, aaron morton <aa...@thelastpickle.com>wrote: > Do you have any ideas, advices on how to investigate in more details? > > > What sort of read queries are you sending ? > Are you using large multi gets ? > Are these servers under IO load ? / Is there a lot of compaction going on ? > > Cheers > > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 3/07/2012, at 3:59 AM, Olivier Mallassi wrote: > > Thanks for the answer. In fact this is maybe simpler. > > We do a lot of inserts and a couple of reads. > - tpstats show us the pending read stage was growing. > - *It seems* there was an issue with one of our request. We have 4 > indexed columns (UT8Type) > First, one of the indexed value was "-----------". we replaced this value > by and empty string and it works better. (running the query in CLI) > > Yet, we still observed pending Read Stage after 10 minutes of run (on a > couple of nodes). > > Do you have any ideas, advices on how to investigate in more details? > > Any help would be appreciate. > > > On Sun, Jul 1, 2012 at 8:56 AM, Jonathan Ellis <jbel...@gmail.com> wrote: > >> Sounds like http://wiki.apache.org/cassandra/FAQ#ubuntu_ec2_hangs to me. >> >> On Fri, Jun 29, 2012 at 1:45 AM, Olivier Mallassi <omalla...@octo.com> >> wrote: >> > Hi all >> > >> > We have a 12 servers clusters (8 cores by machines..). >> > OS is Ubuntu 10.04.2. >> > >> > On one of the machine (only one) and without any load (no inserts, no >> > reads....), we have a huge CPU Load whereas there is no activities (no >> > compaction in progress etc...) >> > A top on the machine show us the process jscv is using all the available >> > CPUs. >> > >> > Is that link to JNA? do you have any ideas? >> > >> > Cheers >> > >> > -- >> > ............................................................ >> > Olivier Mallassi >> > OCTO Technology >> > ............................................................ >> > 50, Avenue des Champs-Elysées >> > 75008 Paris >> > >> > Mobile: (33) 6 28 70 26 61 >> > Tél: (33) 1 58 56 10 00 >> > Fax: (33) 1 58 56 10 01 >> > >> > http://www.octo.com >> > Octo Talks! http://blog.octo.com >> > >> > >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of DataStax, the source for professional Cassandra support >> http://www.datastax.com >> > > > > -- > ............................................................ > Olivier Mallassi > OCTO Technology > ............................................................ > 50, Avenue des Champs-Elysées > 75008 Paris > > Mobile: (33) 6 28 70 26 61 > Tél: (33) 1 58 56 10 00 > Fax: (33) 1 58 56 10 01 > > http://www.octo.com > Octo Talks! http://blog.octo.com > > > > -- ............................................................ Olivier Mallassi OCTO Technology ............................................................ 50, Avenue des Champs-Elysées 75008 Paris Mobile: (33) 6 28 70 26 61 Tél: (33) 1 58 56 10 00 Fax: (33) 1 58 56 10 01 http://www.octo.com Octo Talks! http://blog.octo.com
