Re: Using a separate commit log drive was 4x slower

2010-08-10 Thread Peter Schuller
I have no explanation for the slower reads, but I have an hypothesis on the writes. Your iostat shows: > Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s avgrq-sz > avgqu-sz   await  svctm  %util > cciss/c0d0    0.00   908.50    0.00  110.50 0.00  8152.00    > 73.77

Re: non blocking Cassandra with Tornado

2010-08-10 Thread Ryan Daum
> > >> Barring this we (place where I work, Chango) will probably eventually fork >> Cassandra to have a RESTful interface and use the Jetty async HTTP client to >> connect to it. It's just ridiculous for us to have threads and associated >> resources tied up on I/O-blocked operations. >> > > We've

Re: Growing commit log directory.

2010-08-10 Thread Jonathan Ellis
having 3 digit pending counts in both RRS and RMS is a danger sign. It looks like you are i/o bound on reads, and possibly on writes as well. (commitlog not on separate disk?) On Mon, Aug 9, 2010 at 10:53 PM, Edward Capriolo wrote: > On Mon, Aug 9, 2010 at 8:20 PM, Jonathan Ellis wrote: >> what

Re: COMMIT-LOG_WRITER Assertion Error

2010-08-10 Thread Jonathan Ellis
Can you create a ticket for this? On Mon, Aug 9, 2010 at 8:42 PM, Arya Goudarzi wrote: > I've never run 0.6. I have been running of trunc with automatic svn update > and build everyday at 2pm. One of my nodes got this error which lead to the > same last error prior to build and restart today. H

why does it take 60-90 seconds for a new node to get up?

2010-08-10 Thread S Ahmed
Why is it that, if you set AutoBootStrap = false that it takes 60-90 seconds for the node to announce itself? I just want to understand what is going on during that time, and why that specific timeframe (if there is a reason?)

Re: why does it take 60-90 seconds for a new node to get up?

2010-08-10 Thread Edward Capriolo
On Tue, Aug 10, 2010 at 10:17 AM, S Ahmed wrote: > Why is it that, if you set AutoBootStrap = false that it takes 60-90 seconds > for the node to announce itself? > I just want to understand what is going on during that time, and why that > specific timeframe (if there is a reason?) The bootstrap

Re: explanation of generated files and ops

2010-08-10 Thread Peter Schuller
> Is this pretty much all the files that Cassandra generates? (have I missed > any) I believe so. There will also be some temporary files (*-tmp*) during compaction, and you'll see some *.Compacted marker files (empty IIRC). > What exactly is stored in the "-Filter.db" files? Those are bloom

Re: COMMIT-LOG_WRITER Assertion Error

2010-08-10 Thread Arya Goudarzi
Sure. https://issues.apache.org/jira/browse/CASSANDRA-1376 - Original Message - From: "Jonathan Ellis" To: user@cassandra.apache.org Sent: Tuesday, August 10, 2010 7:05:31 AM Subject: Re: COMMIT-LOG_WRITER Assertion Error Can you create a ticket for this? On Mon, Aug 9, 2010 at 8:42 PM,

Re: Using a separate commit log drive was 4x slower

2010-08-10 Thread Jeremy Davis
Yeah, it has a BBU, and it is charged and on.. Very odd behavior, I'm stumped. -JD On Tue, Aug 10, 2010 at 12:28 AM, Peter Schuller < peter.schul...@infidyne.com> wrote: > I have no explanation for the slower reads, but I have an hypothesis > on the writes. > > Your iostat shows: > > > Device:

Re: Using a separate commit log drive was 4x slower

2010-08-10 Thread Jonathan Ellis
Other activity, e.g. syslog? Journaling at the FS level? you could try making a small partition formatted as ext2. On Tue, Aug 10, 2010 at 5:10 PM, Jeremy Davis wrote: > Yeah, it has a BBU, and it is charged and on.. > Very odd behavior, I'm stumped. > > -JD > > On Tue, Aug 10, 2010 at 12:28 AM,

Re: Using a separate commit log drive was 4x slower

2010-08-10 Thread Peter Schuller
> Yeah, it has a BBU, and it is charged and on.. > Very odd behavior, I'm stumped. I advise double-checking raid volume settings and ensuring that policy is truly such that the write cache is used. This may also be a function of kernel driver settings depending on what RAID controller/kernel versi

Re: explanation of generated files and ops

2010-08-10 Thread Robert Coli
On 8/9/10 9:00 PM, S Ahmed wrote: Is this pretty much all the files that Cassandra generates? (have I missed any) If you are running cassandra via the linux init scripts, you are setting "outfile" for jsvc to be : " -outfile /var/log/$NAME/output.log \ " And with : " -errfile "

Soliciting thoughts on possible read optimization

2010-08-10 Thread Arya Asemanfar
I mentioned this today to a couple folks at Cassandra Summit, and thought I'd solicit some more thoughts here. Currently, the read stage includes checking row cache. So if your concurrent reads is N and you have N reads reading from disk, the next read will block until a disk read finishes, even i

Re: explanation of generated files and ops

2010-08-10 Thread Robert Coli
On 8/9/10 9:00 PM, S Ahmed wrote: What exactly is stored in the "-Filter.db" files? Oh, didn't see this line. http://wiki.apache.org/cassandra/ArchitectureOverview " Bloom filter (all keys in data file). A Bloom filter, is a space-efficient probabilistic data structure that is used to test