Re: Read times on Cassandra

2011-09-10 Thread Yang
Thanks , this link is very useful. to add to that, I just found that the "sar -g " command is very helpful, it shows paging requests. On Thu, Sep 8, 2011 at 9:02 AM, Jonathan Ellis wrote: > What is your bottleneck? > > spyced.blogspot.com/2010/01/linux-performance-basics.html > > On Thu, Sep 8,

Re: Disabling hinted handoff doesn't work in 0.8.4?

2011-09-10 Thread Jeremy Hanna
Turned out that wasn't a problem - I put some notes on the ticket. On Sep 10, 2011, at 6:22 PM, Jeremy Hanna wrote: > I tried looking through the source to see if the log statements would happen > regardless but it doesn't look like it. Also I looked at one of the nodes > via jmx and checked o

Re: Disabling hinted handoff doesn't work in 0.8.4?

2011-09-10 Thread Jeremy Hanna
I tried looking through the source to see if the log statements would happen regardless but it doesn't look like it. Also I looked at one of the nodes via jmx and checked out the System.HintsColumnFamily and there is activity there. So I created a ticket: https://issues.apache.org/jira/browse/

Disabling hinted handoff doesn't work in 0.8.4?

2011-09-10 Thread Jeremy Hanna
We just tried to disable hinted handoff by setting: hinted_handoff_enabled: false in all the nodes of our cluster and restarting them. When they come back up, we continue to see things like this: INFO [HintedHandoff:1] 2011-09-10 22:41:40,813 HintedHandOffManager.java (line 323) Started hinted h

Re: Massive writes when only reading from Cassandra

2011-09-10 Thread Jeremy Hanna
Thanks for the insights. I may first try disabling hinted handoff for one run of our data pipeline and see if it exhibits the same behavior. Will post back if I see anything enlightening there. On Sep 10, 2011, at 5:04 PM, Chris Goffinet wrote: > You could tail the commit log with `strings` t

Re: Massive writes when only reading from Cassandra

2011-09-10 Thread Chris Goffinet
You could tail the commit log with `strings` to see what keys are being inserted. On Sat, Sep 10, 2011 at 2:24 PM, Jonathan Ellis wrote: > Two possibilities: > > 1) Hinted handoff (this will show up in the logs on the sending > machine, on the receiving one it will just look like any other write

Re: Ec2Snitch nodetool issue after upgrade to 0.8.5

2011-09-10 Thread Sasha Dolgy
Of course. Hoping one day I create an issue related to Ec2 that CAN be reproduced... https://issues.apache.org/jira/browse/CASSANDRA-3175 On Sat, Sep 10, 2011 at 10:10 PM, Jonathan Ellis wrote: > Can you create a Jira ticket?

Re: Massive writes when only reading from Cassandra

2011-09-10 Thread Jonathan Ellis
Two possibilities: 1) Hinted handoff (this will show up in the logs on the sending machine, on the receiving one it will just look like any other write) 2) You have something doing writes that you're not aware of, I guess you could track that down using wireshark to see where the write messages a

Re: JNA fails to load ?

2011-09-10 Thread Yang
ok, found it's due to yourkit instrumentation: http://forums.yourkit.com/viewtopic.php?f=3&t=3299 On Fri, Sep 9, 2011 at 5:50 PM, Yang wrote: > I have jna-3.2.7 in the classpath, but at the start of server messages, I see: > > INFO 17:48:12,321 Unable to link C library. Native methods will be di

Re: Massive writes when only reading from Cassandra

2011-09-10 Thread Jeremy Hanna
Oh and we're running 0.8.4 and the RF is 3. On Sep 10, 2011, at 3:49 PM, Jeremy Hanna wrote: > In addition, the mutation stage and the read stage are backed up like: > > Pool NameActive Pending Blocked > ReadStage32 773 0 > RequestRes

Re: Massive writes when only reading from Cassandra

2011-09-10 Thread Jeremy Hanna
In addition, the mutation stage and the read stage are backed up like: Pool NameActive Pending Blocked ReadStage32 773 0 RequestResponseStage 0 0 0 ReadRepairStage 0 0 0 Mut

Massive writes when only reading from Cassandra

2011-09-10 Thread Jeremy Hanna
We are experiencing massive writes to column families when only doing reads from Cassandra. A set of 5 hadoop jobs are reading from Cassandra and then writing out to hdfs. That is the only thing operating on the cluster. We are reading at CL.QUORUM with hadoop and have written with CL.QUORUM.

Re: Ec2Snitch nodetool issue after upgrade to 0.8.5

2011-09-10 Thread Jonathan Ellis
Can you create a Jira ticket? On Sat, Sep 10, 2011 at 6:52 AM, Sasha Dolgy wrote: > maybe it's related to this ... > https://issues.apache.org/jira/browse/CASSANDRA-3114 > > odd thing is, we haven't "moved" to Ec2Snitch ... been using it for > quite a long time now ... > > On Sat, Sep 10, 2011 at

Re: mysterious data disappearance - what happened?

2011-09-10 Thread Yiming Sun
Hi Peter, Good call! I went and checked the seed, and indeed it left unchanged when we copied the config yaml file from the first node to the second node. Thanks! -- Y. On Fri, Sep 9, 2011 at 7:47 PM, Peter Schuller wrote: > > cluster name for both machines. So in other words, if we want to l

Re: shutdown by KILL

2011-09-10 Thread Chris Goffinet
For things like rolling restarts, we do: disablethrift disablegossip (...wait for all nodes to see this node go down..) drain 2011/9/10 Radim Kolar > what is recommended node stop method. drain or kill Java process? i haven't > seen anybody using drain in stop scripts yet > > If i kill Java pro

shutdown by KILL

2011-09-10 Thread Radim Kolar
what is recommended node stop method. drain or kill Java process? i haven't seen anybody using drain in stop scripts yet If i kill Java process and log flush time is set to 10 seconds, will be last 10 seconds lost?

Re: Ec2Snitch nodetool issue after upgrade to 0.8.5

2011-09-10 Thread Sasha Dolgy
maybe it's related to this ... https://issues.apache.org/jira/browse/CASSANDRA-3114 odd thing is, we haven't "moved" to Ec2Snitch ... been using it for quite a long time now ... On Sat, Sep 10, 2011 at 1:42 PM, Sasha Dolgy wrote: > Upgraded one ring that has four nodes from 0.8.0 to 0.8.5 with o

Ec2Snitch nodetool issue after upgrade to 0.8.5

2011-09-10 Thread Sasha Dolgy
Upgraded one ring that has four nodes from 0.8.0 to 0.8.5 with only one minor problem.  It relates to Ec2Snitch when running a 'nodetool ring' from two of the four nodes.  the rest are all working fine: Address         DC          Rack        Status State   Load Owns    Token        14836224792726

Re: Mutations during selects

2011-09-10 Thread Jonathan Ellis
Cassandra makes no attempt to provide cross-row isolation or atomicity. (Within a row, it technically provides atomicity but not isolation.) On Fri, Sep 9, 2011 at 9:33 PM, wrote: > I a thinking about a scenario that goes like this: a node is reading a > secondary index to reply to a select que

Re: Key Range

2011-09-10 Thread Jonathan Ellis
W=Q R=1 is not guaranteed to be consistent, no. you need R=W=Q for that. On Fri, Sep 9, 2011 at 6:59 PM, Alaa Zubaidi wrote: > Hi, > Using 0.7.4, BOP, and RF=3, Write->QUORUM and Read->ONE "NO UPDATES" > How does get_range_slices using Key Range works? Is it possible to get out > of date list of

Re: Is sstable2json broken in 0.8.4+?

2011-09-10 Thread Jonathan Ellis
Cassandra ensures that data you put into a column matches its definition, but if you change its definition later it assumes you know what you're doing and doesn't scan the entire data set to revalidate. You probably need to switch back to bytestype until you can fix that. On Fri, Sep 9, 2011 at 6