Re: Massive writes when only reading from Cassandra

2011-10-21 Thread Jonathan Ellis
Correct. On Fri, Oct 21, 2011 at 6:47 AM, Jeremiah Jordan wrote: > I could be totally wrong here, but If you are doing a QUORUM read and there > is a bad value encountered from the QUORUM won't a repair happen?  I thought > read_repair_chance 0 just means it won't query extra nodes to check for

Re: Massive writes when only reading from Cassandra

2011-10-21 Thread Jeremiah Jordan
I could be totally wrong here, but If you are doing a QUORUM read and there is a bad value encountered from the QUORUM won't a repair happen? I thought read_repair_chance 0 just means it won't query extra nodes to check for bad values. -Jeremiah On Oct 17, 2011, at 4:22 PM, Jeremy Hanna wrote

Re: Massive writes when only reading from Cassandra

2011-10-17 Thread Jeremy Hanna
Even after disabling hinted handoff and setting read_repair_chance to 0 on all our column families, we were still experiencing massive writes. Apparently the read_repair_chance is completely ignored at any CL higher than CL.ONE. So we were doing CL.QUORUM on reads and writes and seeing massive

Re: Massive writes when only reading from Cassandra

2011-09-10 Thread Jeremy Hanna
Thanks for the insights. I may first try disabling hinted handoff for one run of our data pipeline and see if it exhibits the same behavior. Will post back if I see anything enlightening there. On Sep 10, 2011, at 5:04 PM, Chris Goffinet wrote: > You could tail the commit log with `strings` t

Re: Massive writes when only reading from Cassandra

2011-09-10 Thread Chris Goffinet
You could tail the commit log with `strings` to see what keys are being inserted. On Sat, Sep 10, 2011 at 2:24 PM, Jonathan Ellis wrote: > Two possibilities: > > 1) Hinted handoff (this will show up in the logs on the sending > machine, on the receiving one it will just look like any other write

Re: Massive writes when only reading from Cassandra

2011-09-10 Thread Jonathan Ellis
Two possibilities: 1) Hinted handoff (this will show up in the logs on the sending machine, on the receiving one it will just look like any other write) 2) You have something doing writes that you're not aware of, I guess you could track that down using wireshark to see where the write messages a

Re: Massive writes when only reading from Cassandra

2011-09-10 Thread Jeremy Hanna
Oh and we're running 0.8.4 and the RF is 3. On Sep 10, 2011, at 3:49 PM, Jeremy Hanna wrote: > In addition, the mutation stage and the read stage are backed up like: > > Pool NameActive Pending Blocked > ReadStage32 773 0 > RequestRes

Re: Massive writes when only reading from Cassandra

2011-09-10 Thread Jeremy Hanna
In addition, the mutation stage and the read stage are backed up like: Pool NameActive Pending Blocked ReadStage32 773 0 RequestResponseStage 0 0 0 ReadRepairStage 0 0 0 Mut

Massive writes when only reading from Cassandra

2011-09-10 Thread Jeremy Hanna
We are experiencing massive writes to column families when only doing reads from Cassandra. A set of 5 hadoop jobs are reading from Cassandra and then writing out to hdfs. That is the only thing operating on the cluster. We are reading at CL.QUORUM with hadoop and have written with CL.QUORUM.