Read latency skyrockets after 95-98th percentile

hey list, i've been trying to understand why we are seeing rather nasty read latency peaks in as much as 2% of our total read ( not sure what the underlying thrift call is, should be get_slice ) requests that we have been **unable to tie to garbage collection or blocking I/O.** This is what i m

Re: Read latency skyrockets after 95-98th percentile

you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Andras Szerdahelyi [mailto:andras.szerdahe...@ignitionone.com] Sent: Tuesday, July 31, 2012 15:53 To: user@cassandra.apache.org<mailto:user@cassandr

row cache re-fill very slow

me, filter rows that should be in row cache ) at work and this operation is purely disk i/o bound ? ( Admittedly whatever is going on is still much more preferable to starting with a cold row cache ) thanks! Andras Andras Szerdahelyi Solutions Architect, IgnitionOne | 1831 Diegem E.Mommaerts

Re: row cache re-fill very slow

help you find a happy half way point. If i can keep that high enough, with my data retention requirements, save for the absolute first get on a row, i can operate entirely out of memory. thanks! Andras Andras Szerdahelyi Solutions Architect, IgnitionOne | 1831 Diegem E.Mommaertslaan 20A M: +32 493

Re: row cache re-fill very slow

hould be easy to calculate though ) It's been working out great. Andras Szerdahelyi Solutions Architect, IgnitionOne | 1831 Diegem E.Mommaertslaan 20A M: +32 493 05 50 88 | Skype: sandrew84 [cid:7BDF7228-D831-4D98-967A-BE04FEB17544] On 19 Nov 2012, at 22:23, Wei Zhu mailto:wz1...

Re: key cache size in mb = 0 doesn't work?

SSTables in http://en.wikipedia.org/wiki/Page_cache maybe? How many rows do you have in this CF? Are you getting all columns? What do the cassandra.db mbeans say ( hit ratio, cache requests, items etc ) regards, Andras On 27 Nov 2012, at 04:12, Yiming Sun wrote: > Hi, > > I am carrying out s

Re: key cache size in mb = 0 doesn't work?

servers behind some strict university controlled firewall, so it wouldn't be easy for me to tap in using JConsole. Thanks. -- Y. On Mon, Nov 26, 2012 at 11:06 PM, Andras Szerdahelyi mailto:andras.szerdahe...@ignitionone.com>> wrote: SSTables in http://en.wikipedia.org/wiki/Page

entire range of node out of sync -- out of the blue

#x27;t let this finish ) - how do i clean up my stables ( grew from 6k to 20k since this started, while i shut writes off completely ) thanks, Andras Andras Szerdahelyi Solutions Architect, IgnitionOne | 1831 Diegem E.Mommaertslaan 20A M: +32 493 05 50 88 | Skype: sandrew84 [cid:7BDF7228-D831-4D98-967A-BE04FEB17544] <>

Re: entire range of node out of sync -- out of the blue

Thanks! i'm also thinking a repair run without -pr could have caused this maybe ? Andras Szerdahelyi Solutions Architect, IgnitionOne | 1831 Diegem E.Mommaertslaan 20A M: +32 493 05 50 88 | Skype: sandrew84 [cid:7BDF7228-D831-4D98-967A-BE04FEB17544] On 06 Dec 2012, at 04:05, aaron m

Re: entire range of node out of sync -- out of the blue

think this was a repair without -pr thanks, Andras Andras Szerdahelyi Solutions Architect, IgnitionOne | 1831 Diegem E.Mommaertslaan 20A M: +32 493 05 50 88 | Skype: sandrew84 [cid:7BDF7228-D831-4D98-967A-BE04FEB17544] On 18 Dec 2012, at 22:09, B. Todd Burruss mailto:bto...@gmail.com

Re: Changing rpc_port in cassandra.yaml has no effect

log4j.appender.cassandra.layout.ConversionPattern = %d %p %c - %m%n Andras Szerdahelyi Solutions Architect, IgnitionOne | 1831 Diegem E.Mommaertslaan 20A M: +32 493 05 50 88 | Skype: sandrew84 [cid:7BDF7228-D831-4D98-967A-BE04FEB17544] On 25 Dec 2012, at 22:56, Bob Futrelle mailto:bob.futre

Re: "Heap is N.N full." Immediately on startup

//www.mail-archive.com/user@cassandra.apache.org/msg25762.html Cheers - Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 20/02/2013, at 11:27 PM, Andras Szerdahelyi mailto:andras.szerdahe...@ignitionone.com>> wrote: Hey list, Any ideas ( bef

Re: "Heap is N.N full." Immediately on startup

on Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 22/02/2013, at 8:37 AM, Andras Szerdahelyi mailto:andras.szerdahe...@ignitionone.com>> wrote: Thank you- indeed my index interval is 64 with a CF of 300M rows + bloom filter false positive chance

Re: "Heap is N.N full." Immediately on startup

rton http://www.thelastpickle.com - Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 26/02/2013, at 3:22 AM, Andras Szerdahelyi mailto:andras.szerdahe...@ignitionone.com>> wrote: ( I am monitoring via a visual vm plugin

Re: significant NICE cpu usage

Hello, AFAIK Compaction threads run with a lower affinity, I believe that will show up as "niced".. Regards, Andras From: Ian Rose mailto:ianr...@fullstory.com>> Reply-To: user mailto:user@cassandra.apache.org>> Date: Wednesday 8 October 2014 17:29 To: user mailto:user@cassandra.apache.org>> Su

Re: Cassandra slow on some reads

Is row cache enabled on this CF? Try disabling it. Seems like you might have a very wide row there. Can you grep for GCInspector in your Cassandra log? 24G might be a bit too much for the Cassandra JVM, bogging down GC, and not leaving much to page cache ( 32G -24G - Tomcat ). I don’t quite und

Re: repair never finishing 1.0.7

The DCs are communicating over a gateway where I do NAT for ports 7000, 9160 and 7199. Ah, that sounds familiar. You don't mention if you are VPN'd or not. I'll assume you are not. So, your nodes are behind network address translation - is that to say they advertise ( broadcast ) their inter

Re: repair never finishing 1.0.7

T sending the packets on the other side to the real destination IP, having replaced the source IP with the initial sender's IP (at least in my understanding of it). What might be the problem given the configuration? How to fix this? Cheers, Alex On Mon, Jun 25, 2012 at 12:47 PM, Andras Sze

Re: 33million hinted handoffs from nowhere

--- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 15/03/2013, at 2:30 AM, Andras Szerdahelyi mailto:andras.szerdahe...@ignitionone.com>> wrote: ( The previous letter was sent prematurely, sorry. ) This node is the only

Re: Cassandra freezes

I'd say GC. Please fill in form CASS-FREEZE-001 below and get back to us :-) ( sorry ) How big is your JVM heap ? How many CPUs ? Garbage collection taking long ? ( look for log lines from GCInspector) Running out of heap ? ( "heap is .. full" log lines ) Any tasks backing up / being dropped ? (

Re: Cassandra freezes

. Upping index_interval from 128 to 512 (this seemed to reduce our memory >usage significantly!!!) >3. Just add more nodes as moving the rows to other servers reduces memory >from #1 and #2 above since the server would have less rows > >Later, >Dean > >On 3/20/13 6:29 AM,

Re: index_interval memory savings in our case(if you are curious)Š (and performance result)...

nodes hitting RAM limit and slowing down >causing website to get real slow). > >Dean > >On 3/20/13 6:41 AM, "Andras Szerdahelyi" > wrote: > >>2. Upping index_interval from 128 to 512 (this seemed to reduce our >>memory >>usage significantly!!!) &g

Re: index_interval memory savings in our case(if you are curious)Š (and performance result)...

nterval >change was working so we dug into the code to find out, it basically seems >to immediately convert on startup though doesn't log anything except at a >"debug" level which we don't have on. > >Dean > > > >On 3/20/13 6:58 AM, "Andras Szerdahely

Re: Cassandra freezes

I can not find the reference that notes having to upgradesstables when you change this. I really hope such complex assumptions are not formulating in my head just on their own and there actually exists some kind of reliable reference that clears this up :-) but, # index_interval controls the samp

Re: Cassandra freezes

.org/jira/browse/CASSANDRA-2392). That being said, if the index_interval value used by a summary saved doesn't match the current one while the sstable is loaded, the summary is recomputed anyway, so restarting a node should always take a new index_interval setting into account. -- Sylvai

Re: 33million hinted handoffs from nowhere

/src/java/org/apache/cassand >ra/gms/GossiperMBean.java#L28 > >Cheers > >- >Aaron Morton >Freelance Cassandra Consultant >New Zealand > >@aaronmorton >http://www.thelastpickle.com > >On 20/03/2013, at 11:10 PM, Andras Szerdahelyi > wrote: >

bloom filter fp ratio of 0.98 with fp_chance of 0.01

Hello list, Could anyone shed some light on how an FP chance of 0.01 coexist with a measured FP ratio of .. 0.98 ? Am I reading this wrong or are 98% of the requests hitting the bloom filter create a false positive while the "target" false ratio is 0.01? ( Also key cache hit ratio is around 0.0

Re: bloom filter fp ratio of 0.98 with fp_chance of 0.01

o when consdiering the sstable size. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 26/03/2013, at 6:16 AM, Andras Szerdahelyi mailto:andras.szerdahe...@ignitionone.com>> wrote: Hello list, Could anyone shed

Re: multiple Datacenter values in PropertyFileSnitch

I would replicate your different keyspaces to different DCs and scale those appropriately So, for example, HighLoad KS replicates to really-huge-dc, which would have, 10 nodes, LowerLoad KS replicates to smaller-dc with 5 nodes. The idea is , you do not mix your different keyspaces in the same da