Re: High read latency after data volume increased

2015-01-12 Thread Jonathan Lacefield
There's likely 2 things occurring 1) the cfhistograms error is due to https://issues.apache.org/jira/browse/CASSANDRA-8028 Which is resolved in 2.1.3. Looks like voting is under way for 2.1.3. As rcoli mentioned, you are running the latest open source of C* which should be treated as beta until a

Re: High read latency after data volume increased

2015-01-09 Thread Roni Balthazar
Hi there, The compaction remains running with our workload. We are using SATA HDDs RAIDs. When trying to run cfhistograms on our user_data table, we are getting this message: nodetool: Unable to compute when histogram overflowed Please see what happens when running some queries on this cf: http:

Re: High read latency after data volume increased

2015-01-09 Thread datastax
Hello You may not be experiencing versioning issues. Do you know if compaction is keeping up with your workload? The behavior described in the subject is typically associated with compaction falling behind or having a suboptimal compaction strategy configured. What does the output of nod

RE: High read latency after data volume increased

2015-01-09 Thread Jason Kushmaul | WDA
an Tarbox [mailto:briantar...@gmail.com] Sent: Friday, January 9, 2015 8:56 AM To: user@cassandra.apache.org Subject: Re: High read latency after data volume increased C* seems to have more than its share of "version x doesn't work, use version y " type issues On Thu, Jan 8, 2015

Re: High read latency after data volume increased

2015-01-09 Thread Brian Tarbox
C* seems to have more than its share of "version x doesn't work, use version y " type issues On Thu, Jan 8, 2015 at 2:23 PM, Robert Coli wrote: > On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar > wrote: > >> We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. >> > > https://eng

Re: High read latency after data volume increased

2015-01-08 Thread Robert Coli
On Thu, Jan 8, 2015 at 6:38 PM, Roni Balthazar wrote: > We downgraded to 2.1.1, but got the very same result. The read latency is > still high, but we figured out that it happens only using a specific > keyspace. > Note that downgrading is officially unsupported, but is probably safe between tho

Re: High read latency after data volume increased

2015-01-08 Thread Roni Balthazar
Hi Robert, We downgraded to 2.1.1, but got the very same result. The read latency is still high, but we figured out that it happens only using a specific keyspace. Please see the graphs below... ​ Trying another keyspace with 600+ reads/sec, we are getting the acceptable ~30ms read latency. Let

Re: High read latency after data volume increased

2015-01-08 Thread Robert Coli
On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar wrote: > We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. > https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ 2.1.2 in particular is known to have significant issues. You'd be better off running 2.1.1 ...

High read latency after data volume increased

2015-01-08 Thread Roni Balthazar
Hi there, We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. While our data volume is increasing (34 TB now), we are running into some problems: 1) Read latency is around 1000 ms when running 600 reads/sec (DC1 CL.LOCAL_ONE). At the same time the load average is about 20-30 on all