Re: heavy insert load overloads CPUs, with MutationStage pending

tityCompositeColumn(sub.getEntityType().getName(), > sub.getEntityId(), > "indexed_tenant_id"), tenantId); > } > > Hope that helps, > Paul > > > From: Keith Freeman [mailto:8fo...@gmail.com] > Sent: Thursday, September 12, 2013 12:

Re: heavy insert load overloads CPUs, with MutationStage pending

Also, I was working on this a bit for a client so compiled my notes and approach into a blog post for posterity (and so it's easier to find for others): http://thelastpickle.com/blog/2013/09/13/CQL3-to-Astyanax-Compatibility.html Paul's method on this thread is cited at the bottom as well. On Fr

Re: heavy insert load overloads CPUs, with MutationStage pending

https://github.com/Netflix/astyanax/issues/391 I've gotten in touch with a couple of netflix folks and they are going to try to roll a release shortly. You should be able to build against 1.2.2 and 'talking' to 1.2.9 instance should work. Just a PITA development wise to maintain a different versi

Re: heavy insert load overloads CPUs, with MutationStage pending

Paul- Sorry to go off-list but I'm diving pretty far into details here. Ignore if you wish. Thanks a lot for the example, definitely very helpful. I'm surprised that the Cassandra experts aren't more interested-in/alarmed-by our results, it seems like we've proved that insert performance fo

Re: heavy insert load overloads CPUs, with MutationStage pending

@gmail.com] Sent: Wednesday, September 11, 2013 1:06 PM To: user@cassandra.apache.org Subject: Re: heavy insert load overloads CPUs, with MutationStage pending Thanks, I had seen your stackoverflow post. I've got hundreds of (wide-) rows, and the writes are pretty well distributed across them.

RE: heavy insert load overloads CPUs, with MutationStage pending

Freeman [mailto:8fo...@gmail.com] Sent: Thursday, September 12, 2013 12:10 PM To: user@cassandra.apache.org Subject: Re: heavy insert load overloads CPUs, with MutationStage pending Ok, your results are pretty impressive, I'm giving it a try.  I've made some initial attempts to use A

Re: heavy insert load overloads CPUs, with MutationStage pending

I have the defaults as shown in your response. On 09/10/2013 01:59 PM, sankalp kohli wrote: What have you set these to? # commitlog_sync may be either "periodic" or "batch." # When in batch mode, Cassandra won't ack writes until the commit log # has been fsynced to disk. It will wait up to # co

RE: heavy insert load overloads CPUs, with MutationStage pending

using the thrift batch_mutate to write a full > wide-row at once instead of using many CQL INSERT statements. > > > > -Paul > > > >> -Original Message- > >> From: Keith Freeman [mailto:8fo...@gmail.com] > >> Sent: Wednesday, September 11, 2013 9:1

Re: heavy insert load overloads CPUs, with MutationStage pending

Freeman [mailto:8fo...@gmail.com] Sent: Wednesday, September 11, 2013 9:16 AM To:user@cassandra.apache.org Subject: Re: heavy insert load overloads CPUs, with MutationStage pending On 09/10/2013 11:42 AM, Nate McCall wrote: With SSDs, you can turn up memtable_flush_writers - try 3 initially (1 by d

RE: heavy insert load overloads CPUs, with MutationStage pending

-Paul > -Original Message- > From: Keith Freeman [mailto:8fo...@gmail.com] > Sent: Wednesday, September 11, 2013 9:16 AM > To: user@cassandra.apache.org > Subject: Re: heavy insert load overloads CPUs, with MutationStage pending > > > On 09/10/2013 11:42 AM, Nate McC

Re: heavy insert load overloads CPUs, with MutationStage pending

On 09/10/2013 11:42 AM, Nate McCall wrote: With SSDs, you can turn up memtable_flush_writers - try 3 initially (1 by default) and see what happens. However, given that there are no entries in 'All time blocked' for such, they may be something else. Tried that, it seems to have reduced the loads

Re: heavy insert load overloads CPUs, with MutationStage pending

I have RF=2 On 09/10/2013 11:18 AM, Robert Coli wrote: On Tue, Sep 10, 2013 at 10:17 AM, Robert Coli > wrote: On Tue, Sep 10, 2013 at 7:55 AM, Keith Freeman <8fo...@gmail.com > wrote: On my 3-node cluster (v1.2.8) with 4-cor

Re: heavy insert load overloads CPUs, with MutationStage pending

What have you set these to? # commitlog_sync may be either "periodic" or "batch." # When in batch mode, Cassandra won't ack writes until the commit log # has been fsynced to disk. It will wait up to # commitlog_sync_batch_window_in_ms milliseconds for other writes, before # performing the sync. #

heavy insert load overloads CPUs, with MutationStage pending

On my 3-node cluster (v1.2.8) with 4-cores each and SSDs for commitlog and data, I get high CPU loads during a heavy-ish wide-row insert load into a single CF (5000 1k inserts/sec), e.g. uptime load avg for last minute 18/11/10. Checking tpstats, I see MutationStage pending on all the nodes, e

Re: heavy insert load overloads CPUs, with MutationStage pending

With SSDs, you can turn up memtable_flush_writers - try 3 initially (1 by default) and see what happens. However, given that there are no entries in 'All time blocked' for such, they may be something else. How are you inserting the data? On Tue, Sep 10, 2013 at 12:40 PM, Keith Freeman <8fo...@gm

Re: heavy insert load overloads CPUs, with MutationStage pending

On 09/10/2013 11:17 AM, Robert Coli wrote: On Tue, Sep 10, 2013 at 7:55 AM, Keith Freeman <8fo...@gmail.com > wrote: On my 3-node cluster (v1.2.8) with 4-cores each and SSDs for commitlog and data On SSD, you don't need to separate commitlog and data. You onl

Re: heavy insert load overloads CPUs, with MutationStage pending

On Tue, Sep 10, 2013 at 7:55 AM, Keith Freeman <8fo...@gmail.com> wrote: > On my 3-node cluster (v1.2.8) with 4-cores each and SSDs for commitlog and > data On SSD, you don't need to separate commitlog and data. You only win from this separation if you have a head to not-move between appends to

Re: heavy insert load overloads CPUs, with MutationStage pending

On Tue, Sep 10, 2013 at 10:17 AM, Robert Coli wrote: > On Tue, Sep 10, 2013 at 7:55 AM, Keith Freeman <8fo...@gmail.com> wrote: > >> On my 3-node cluster (v1.2.8) with 4-cores each and SSDs for commitlog >> and data > > BTW, is RF=3? If so, you effectively have a 1 node cluster while writing. =R