Re: performance with a "large" number of supercolumns/columns

2010-07-07 Thread Terje Marthinussen
Hi, yes, I have been looking a bit on this, but not as much as I wanted to. I kind of forgot about the whole double call on first access (probably something related to caching?) as even fixing that, the performance is not good enough for what I need. The hasNext() call causes interactions with b

Re: Functional tests

2010-07-07 Thread Jonathan Ellis
What errors are you seeing? I run nosetests every day. On Mon, Jun 28, 2010 at 3:39 PM, Marty Greenia wrote: > Hello, > > Are the currently checked in functional tests working? The instructions for > running them (http://wiki.apache.org/cassandra/HowToContribute) are not > working -- nosetests r

Re: performance with a "large" number of supercolumns/columns

2010-07-07 Thread Jonathan Ellis
Hi Terje, Sorry to not get to this sooner. Are you still looking into this? On Tue, Jun 22, 2010 at 12:47 PM, Terje Marthinussen wrote: > Hi, > > I was looking a bit on a case we have with columnfamily which has 439k > supercolumns, each supercolumn with ~3 columns each so a total of some 1.3 >

Re: Minimizing the impact of compaction on latency and throughput

2010-07-07 Thread Jonathan Ellis
On Wed, Jul 7, 2010 at 4:57 PM, Peter Schuller wrote: >> This makes sense, but from what I have seen, read contention vs >> cassandra is a much bigger deal than write contention (meant "read contention vs compaction") > I am not really concerned with write performance, but rather with > writes a

Re: Minimizing the impact of compaction on latency and throughput

2010-07-07 Thread Peter Schuller
> This makes sense, but from what I have seen, read contention vs > cassandra is a much bigger deal than write contention (unless you > don't have a separate device for your commitlog, but optimizing for > that case isn't one of our goals). I am not really concerned with write performance, but rat

Re: Minimizing the impact of compaction on latency and throughput

2010-07-07 Thread Jonathan Ellis
This makes sense, but from what I have seen, read contention vs cassandra is a much bigger deal than write contention (unless you don't have a separate device for your commitlog, but optimizing for that case isn't one of our goals). On Wed, Jul 7, 2010 at 12:09 PM, Peter Schuller wrote: > Hello,

Re: Minimizing the impact of compaction on latency and throughput

2010-07-07 Thread Rishi Bhardwaj
I have done some bulk write performance tests and I saw background compaction making a big detrimental impact on the write performance. I was also wondering if there is a tunable to limit the frequency of the compaction on the sstables. If not, then adding such a configuration option would also

Minimizing the impact of compaction on latency and throughput

2010-07-07 Thread Peter Schuller
Hello, I have repeatedly seen users report that background compaction is overly detrimental to the behavior of the node with respect to latency. While I have not yet deployed cassandra in a production situation where latencies are closely monitored, these reports do not really sound very surprisin

Re: Why so many commitlogs ?

2010-07-07 Thread Anty
yes, i know. I only insert records into one CF. when a memtable flush complete, commitlog will check if there are some obsolete commitlog segments. I don't known why there are so many commitlog file out there. is there a possibility that too many memtables is waiting for flushing, which prevent m

Re: Why so many commitlogs ?

2010-07-07 Thread Jonathan Ellis
commitlogs can be removed after _all_ the CFs they have data for have been flushed. On Wed, Jul 7, 2010 at 5:21 AM, Anty wrote: > Hi:all > In my little cluter ,after i insert many many records into cassandra, there > are hundreds of commit log files in commitlog log directory. > is it normal ? >

Why so many commitlogs ?

2010-07-07 Thread Anty
Hi:all In my little cluter ,after i insert many many records into cassandra, there are hundreds of commit log files in commitlog log directory. is it normal ? I read the source code of commitlog , there shouldn't be so many commitlog log files . any clue will be appreciate. -- Best Regards Anty R