Cassandra 1.0 hangs during GC

This is a cluster of 2 nodes, each having 8G of operating memory, replicationfactor=2 Write/read pressure is quite low and almost never exceeds 10/second >From time to time (2-3 times in a month) I see GC activity in logs and for >this time cassandra stops responding to requests which results i

Re: Cassandra 1.0 hangs during GC

ave a lot of CF's and/or secondary indexes ? > Can you see log lines about the "liveRatio" for your cf's ? > I would upgrade to 1.0.10 before getting too carried away though. > Cheers > > - > Aaron Morton > Freelance Developer > @aaronmorton >

nodetool move causes summary load to grow

I have a testing cluster cassandra 1.1.2 with default memory and cache settings, 1 CF, 1 KS, RF = 2 This is an empty cluster 10.111.1.141datacenter1 rack1 Up Normal 43.04 KB100.00% 0 10.111.1.142datacenter1 rack1

Re: Cassandra 1.0 hangs during GC

pping. Also which jvm do you use? > > kind regards > > Joost > > On Mon, Jul 23, 2012 at 10:07 AM, Nikolay Kоvshov wrote: > >>   21th I have mirgated to cassandra 1.1.2 but see no improvement >> >>  cat /var/log/cassandra/Earth1.log | grep "GC for" >&

Re: Cassandra 1.0 hangs during GC

I ran sar only recently after your advice and did not meet any huge GC-s on that server At 08:14 there was a GC lasting 4.5 seconds, that's not five minutes of course, but also quite an unpleasant value; Still I'm waiting for big GC values and will provide according sar logs. 07:25:01 PM pgp

Fwd: Cassandra 1.0 hangs during GC

13:07, "Joost van de Wijgerd" : >  You are better off using Sun Java 6 to run Cassandra. In the past >  there were issues reported on 7. Can you try running it on Sun Java 6? > >  kind regards > >  Joost > >  On Tue, Jul 24, 2012 at 10:04 AM, Nikolay Kоvshov wrote: &g

Re: Cassandra 1.0 hangs during GC

13:07, "Joost van de Wijgerd" : > You are better off using Sun Java 6 to run Cassandra. In the past > there were issues reported on 7. Can you try running it on Sun Java 6? > > kind regards > > Joost > > On Tue, Jul 24, 2012 at 10:04 AM, Nikolay Kоvshov wrote: > &g

Re: Cassandra 1.0 hangs during GC

java version : same - RAM and CPU : 'bad' cluster has more - Ubuntu version - Networking - What else??? 26.07.2012, 10:25, "Nikolay Kоvshov" : > Sun Java 6 didn't help it at all > > Sar shows no special activity on the long GC times > > And I have r

Re: Cassandra 1.0 hangs during GC

- JNA is not installed on both machines 30.07.2012, 14:44, "Mateusz Korniak" : > On Monday 30 of July 2012, Nikolay Kоvshov wrote: > >>  What I plan to compare between 'bad' cluster and 'good' cluster: >> >>  - Configs, schemas, data etc: s

Re: Cassandra 1.0 hangs during GC

of July 2012, Nikolay Kоvshov wrote: > >>  -  JNA is not installed on both machines > > So your GC times may be strongly [1] affected by swapping. > IIRC, also snapshotting is more expensive and may trigger more swapping. > I would start with turning JNA mlockall on [2]. > &

Re: Cassandra 1.0 hangs during GC

And the final solution http://unbxd.com/blog/2012/07/java-and-ksoftirqd-100-cpu-due-to-leap-second/ Doing $ date -s "`date`" solved the problem. 30.07.2012, 16:09, "Nikolay Kоvshov" : >  You mean using swap memory? I have total of 48G of RAM and Cassandra never >

How to bring cluster to consistency

Hello everybody I have a very simple cluster containing 2 servers. Replication_factor = 2, Consistency_level of reads and writes = 1 10.111.1.141datacenter1 rack1 Up Normal 1.5 TB 100.00% vjpigMzv4KkX3x7z 10.111.1.142datacenter1 rack1 Up Normal 1.41 TB

Re: How to bring cluster to consistency

ame token? The output indicates > that 100% of your key range is assigned to 10.111.1.141 (and > therefore 10.111.1.142 holds replicas only) > 3/ maybe repair is being affected by above, but in my experience it can be > sensitive > > On 13 February 2012 13:06, Nikolay Kоvshov w

OOM on heavy write load

I am using Cassandra 0.7.0 with following settings binary_memtable_throughput_in_mb: 64 in_memory_compaction_limit_in_mb: 64 keys_cached 1 million rows_cached 0 RAM for Cassandra 2 GB I run very simple test 1 Node with 4 HDDs (1 HDD - commitlog and caches, 3 HDDs - data) 1 KS => 1 CF => 1 Colum

Re: OOM on heavy write load

//www.datastax.com/docs/0.7/troubleshooting/index#nodes-are-dying-with-oom-errors > >  On Fri, Apr 22, 2011 at 8:00 AM, Nikolay Kоvshov ;; > wrote: >>   I am using Cassandra 0.7.0 with following settings >> >>   binary_memtable_throughput_in_mb: 64 >>   in_memory

Re: OOM on heavy write load

assandra to guard against OOM, you must configure > nodes such that the max memory usage on each node, that is max size all your > caches and memtables can potentially grow to, is less than your heap size. > > From: Nikolay Kоvshov [nkovs...@