Re: Node Dead/Up

2012-10-23 Thread aaron morton
> I don't see errors in the logs, but I do see > a lot of dropped mutations and reads. Any correlation? Yes. The dropped messages mean the server is overloaded. Look for log messages from the GCInspector in /var/log/cassandra/system.log and/or an overloaded IO system see http://spyced.blogspo

Re: constant CMS GC using CPU time

2012-10-23 Thread B. Todd Burruss
Regarding memory usage after a repair ... Are the merkle trees kept around? On Oct 23, 2012 3:00 PM, "Bryan Talbot" wrote: > On Mon, Oct 22, 2012 at 6:05 PM, aaron morton wrote: > >> The GC was on-going even when the nodes were not compacting or running a >> heavy application load -- even when th

Re: Java 7 support?

2012-10-23 Thread Eric Evans
On Tue, Oct 16, 2012 at 7:54 PM, Rob Coli wrote: > On Tue, Oct 16, 2012 at 4:45 PM, Edward Sargisson > wrote: >> The Datastax documentation says that Java 7 is not recommended[1]. However, >> Java 6 is due to EOL in Feb 2013 so what is the reasoning behind that >> comment? > > I've asked this app

Re: What does ReadRepair exactly do?

2012-10-23 Thread Manu Zhang
I think so. Otherwise, we may never complete a read if writes come in continuously. On Wed, Oct 24, 2012 at 9:04 AM, shankarpnsn wrote: > manuzhang wrote > > why repair again? We block until the consistency constraint is met. Then > > the latest version is returned and repair is done asynchronou

Re: What does ReadRepair exactly do?

2012-10-23 Thread shankarpnsn
manuzhang wrote > why repair again? We block until the consistency constraint is met. Then > the latest version is returned and repair is done asynchronously if any > mismatch. We may retry read if fewer columns than required are returned. Just to make sure I understand you correct, considering th

Re: What does ReadRepair exactly do?

2012-10-23 Thread Manu Zhang
why repair again? We block until the consistency constraint is met. Then the latest version is returned and repair is done asynchronously if any mismatch. We may retry read if fewer columns than required are returned. On Wed, Oct 24, 2012 at 6:10 AM, shankarpnsn wrote: > Hello, > > This conversa

Re: Node Dead/Up

2012-10-23 Thread Jason Hill
thanks for the replies. I'll check the load on the node that is reported as DOWN/UP. At first glace it does not appear to be overloaded. But, I will dig in deeper, is there a specific indicator on an ubuntu server that would be useful to me? Also, I didn't make it clear, but in my original post,

Re: What does ReadRepair exactly do?

2012-10-23 Thread shankarpnsn
Hello, This conversation precisely targets a question that I had been having for a while - would be grateful if you someone cloud clarify it a little further: Considering the case of a "repair" created due to a consistency constraint (first case in the discussion above), would the following int

Re: constant CMS GC using CPU time

2012-10-23 Thread Bryan Talbot
On Mon, Oct 22, 2012 at 6:05 PM, aaron morton wrote: > The GC was on-going even when the nodes were not compacting or running a > heavy application load -- even when the main app was paused constant the GC > continued. > > If you restart a node is the onset of GC activity correlated to some event?

Re: Strange row expiration behavior

2012-10-23 Thread aaron morton
> In the first example, I am running compaction at step 7 through nodetool, Sorry missed that. >> insert a couple rows with ttl=5 (again, just a small number) >> ExpiringColumn's are only purged if their TTL has expired AND their absolute (node local) expiry time occurred before the current "g

Re: What does ReadRepair exactly do?

2012-10-23 Thread Shankaranarayanan P N
Hello, This conversation precisely targets a question that I had been having for a while - would be grateful if you someone cloud clarify it a little further: Considering the case of a "repair" created due to a consistency constraint (first case in the discussion above), would the following inter

Re: nodetool cleanup

2012-10-23 Thread B. Todd Burruss
since SSTABLEs are immutable, it must create new SSTABLEs without the data that the node is no longer a replica for ... but it doesn't remove deleted data. seems like a possible optimization to also removed deleted data and tombstone cleanup ... but i guess cleanup shouldn't really be used that mu

Re: constant CMS GC using CPU time

2012-10-23 Thread Bryan Talbot
These GC settings are the default (recommended?) settings from cassandra-env. I added the UseCompressedOops. -Bryan On Mon, Oct 22, 2012 at 6:15 PM, Will @ SOHO wrote: > On 10/22/2012 09:05 PM, aaron morton wrote: > > # GC tuning options > JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC" > JVM_OPTS="$J

Re: Strange row expiration behavior

2012-10-23 Thread Stephen Mullins
Thanks Aaron, my reply is inline below: On Tue, Oct 23, 2012 at 2:38 AM, aaron morton wrote: > Performing these steps results in the rows still being present using > *cassandra-cli > list*. > > I assume you are saying the row key is listed without any columns. aka a > ghost row. > Correct. > >

Re: Node Dead/Up

2012-10-23 Thread aaron morton
> check 10.50.10.21 for what is the system load. +1 And take a look in the logs on 10.21. 10.21 is being seen as down by the other nodes. it could be: * 10.21 failing to gossip fast enough, say by being overloaded to in long ParNew GC pauses. * This node failing to process gossip fast , say b

Re: How to change the seed node Cassandra 1.0.11

2012-10-23 Thread aaron morton
Just change the yaml and restart. The seed list is not persisted i the System KS (like the token assignment). I would suggest running 2 or 3 seeds in your cluster, even if you only have 3 nodes. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.c

Re: nodetool cleanup

2012-10-23 Thread aaron morton
> what is the internal memory model used? It sounds like it doesn't have a page > manager? Nodetool cleanup is a maintenance process to remove data on disk that the node is no longer a replica for. It is typically used after the token assignments have been changed. Cheers - Aa

Re: Strange row expiration behavior

2012-10-23 Thread aaron morton
> Performing these steps results in the rows still being present using > cassandra-cli list. I assume you are saying the row key is listed without any columns. aka a ghost row. > What gets really odd is if I add these steps it works That's working as designed. gc_grace_seconds does not spec

Re: tuning for read performance

2012-10-23 Thread aaron morton
>> and nodetool tpstats always shows pending tasks in the ReadStage. Are clients reading a single row at a time or multiple rows ? Each row requested in a multi get becomes a task in the read stage. Also look at the type of query you are sending. I talked a little about the performance of diffe

Re: Node Dead/Up

2012-10-23 Thread Jason Wee
check 10.50.10.21 for what is the system load. On Tue, Oct 23, 2012 at 10:41 AM, Jason Hill wrote: > Hello, > > I'm on version 1.0.11. > > I'm seeing this in my system log with occasional frequency: > > INFO [GossipTasks:1] 2012-10-23 02:26:34,449 Gossiper.java (line 818) > InetAddress /10.50.10

Re: What does ReadRepair exactly do?

2012-10-23 Thread aaron morton
Yes, all this starts because of the call to filter.collateColumns()… The ColumnFamily is an implementation of o.a.c.dbAbstractColumnContainer , the methods to add columns on that interface pass through to an implementation of ISortedColumns. The implementations of ISortedColumns, e.g. ArrayBac