Re: Really high read latency

2015-03-23 Thread Ben Bromhead
nodetool cfhistograms is also very helpful in diagnosing these kinds of data modelling issues. On 23 March 2015 at 14:43, Chris Lohfink wrote: > > >> Compacted partition maximum bytes: 36904729268 > > thats huge... 36gb rows are gonna cause a lot of problems, even when you > specify a precise c

Re: Really high read latency

2015-03-23 Thread Chris Lohfink
>> Compacted partition maximum bytes: 36904729268 thats huge... 36gb rows are gonna cause a lot of problems, even when you specify a precise cell under this it still is going to have an enormous column index to deserialize on every read for the partition. As mentioned above, you should include y

Re: Really high read latency

2015-03-23 Thread Dave Galbraith
I haven't deleted anything. Here's output from a traced cqlsh query (I tried to make the spaces line up, hope it's legible): Execute CQL3 query | 2015-03-23 21:04:37.422000 | 172.31.32.211 | 0 Parsing select * from default.metrics where row_time = 16511 and attrs = '[redacted]' limit

Re: Really high read latency

2015-03-23 Thread Eric Stevens
Enable tracing in cqlsh and see how many sstables are being lifted to satisfy the query (are you repeatedly writing to the same partition [row_time]) over time?). Also watch for whether you're hitting a lot of tombstones (are you deleting lots of values in the same partition over time?). On Mon,

Re: Really high read latency

2015-03-23 Thread Dave Galbraith
Duncan: I'm thinking it might be something like that. I'm also seeing just a ton of garbage collection on the box, could it be pulling rows for all 100k attrs for a given row_time into memory since only row_time is the partition key? Jens: I'm not using EBS (although I used to until I read up on h

Re: Really high read latency

2015-03-23 Thread Jens Rantil
Also, two control questions: - Are you using EBS for data storage? It might introduce additional latencies. - Are you doing proper paging when querying the keyspace? Cheers, Jens On Mon, Mar 23, 2015 at 5:56 AM, Dave Galbraith wrote: > Hi! So I've got a table like this: > > CREATE TAB

Re: Really high read latency

2015-03-23 Thread Duncan Sands
Hi Dave, On 23/03/15 05:56, Dave Galbraith wrote: Hi! So I've got a table like this: CREATE TABLE "default".metrics (row_time int,attrs varchar,offset int,value double, PRIMARY KEY(row_time, attrs, offset)) WITH COMPACT STORAGE AND bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND comment

Really high read latency

2015-03-22 Thread Dave Galbraith
Hi! So I've got a table like this: CREATE TABLE "default".metrics (row_time int,attrs varchar,offset int,value double, PRIMARY KEY(row_time, attrs, offset)) WITH COMPACT STORAGE AND bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND comment='' AND dclocal_read_repair_chance=0 AND gc_grace_sec

Re: Noticing really high read latency

2014-03-05 Thread Eric Plowe
Disregard... heh. Was reading the latency as SECONDS. Sorry, it's been one of those weeks. On Wed, Mar 5, 2014 at 1:44 AM, Eric Plowe wrote: > Background info: > > 6 node cluster. > 24 gigs of ram per machine > 8 gigs of ram dedicated to c* > 4 4 core cpu's > 2 250 gig SSD's raid 0 > Running c*

Noticing really high read latency

2014-03-04 Thread Eric Plowe
Background info: 6 node cluster. 24 gigs of ram per machine 8 gigs of ram dedicated to c* 4 4 core cpu's 2 250 gig SSD's raid 0 Running c* 1.2.6 The CF is configured as followed CREATE TABLE behaviors ( uid text, buid int, name text, expires text, value text, PRIMARY KEY (uid, buid,