I see the same behavior with smaller batch sizes. It appears to happen when starting Cassandra with the defaults on relatively large systems. Attached is a script I created to reproduce the problem. (usage: mutate.sh /path/to/apache-cassandra-0.7.3-bin.tar.gz) It extracts a stock cassandra distribution to a temp dir and starts it up, (single node) then creates a keyspace with a column family and does a batch mutate to insert 5000 columns.
When I run it on my laptop (Fedora 14, 64-bit, 4 cores, 8GB RAM) it flushes one Memtable with 5000 operations When I run it on a server (RHEL5, 64-bit, 16 cores, 96GB RAM) it flushes 100 Memtables with anywhere between 1 operation and 359 operations (35 bytes and 12499 bytes)
I'm guessing I can override the JVM memory parameters to avoid the frequent flushing on the server, but I haven't experimented with that yet. The only difference in the effective command line between the laptop and server is "-Xms3932M -Xmx3932M -Xmn400M" on the laptop and "-Xms48334M -Xmx48334M -Xmn1600M" on the server.
-- Erik Forkalsrud Commission Junction On 03/10/2011 09:18 AM, Ryan King wrote:
Why use such a large batch size? -ryan On Thu, Mar 10, 2011 at 6:31 AM, Desimpel, Ignace <ignace.desim...@nuance.com> wrote:Hello, I had a demo application with embedded cassandra version 0.6.x, inserting about 120 K row mutations in one call. In version 0.6.x that usually took about 5 seconds, and I could repeat this step adding each time the same amount of data. Running on a single CPU computer, single hard disk, XP 32 bit OS, 1G memory I tested this again on CentOS 64 bit OS, 6G memory, different settings of memtable_throughput_in_mb and memtable_operations_in_millions. Also tried version 0.7.3. Also the same behavior. Now with version 0.7.2 the call returns with a timeout exception even using a timeout of 120000 (2 minutes). I see the CPU time going to 100%, a lot of disk writing ( giga bytes), a lot of log messages about compacting, flushing, commitlog, … Below you can find some information using the nodetool at start of the batch mutation and also after 14 minutes. The MutationStage is clearly showing how slow the system handles the row mutations. Attached : Cassandra.yaml with at end the description of my database structure using yaml Attached : log file with cassandra output. Any idea what I could be doing wrong? Regards, Ignace Desimpel ignace.desim...@nuance.com At start of the insert (after inserting 124360 row mutations) I get the following info from the nodetool : C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com info Starting NodeTool 34035877798200531112672274220979640561 Gossip active : true Load : 5.49 MB Generation No : 1299502115 Uptime (seconds) : 1152 Heap Memory (MB) : 179,84 / 1196,81 C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com tpstats Starting NodeTool Pool Name Active Pending Completed ReadStage 0 0 40637 RequestResponseStage 0 0 30 MutationStage 32 121679 72149 GossipStage 0 0 0 AntiEntropyStage 0 0 0 MigrationStage 0 0 1 MemtablePostFlusher 0 0 6 StreamStage 0 0 0 FlushWriter 0 0 5 MiscStage 0 0 0 FlushSorter 0 0 0 InternalResponseStage 0 0 0 HintedHandoff 0 0 0 After 14 minutes (timeout exception after 2 minutes : see log file) I get : C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com info Starting NodeTool 34035877798200531112672274220979640561 Gossip active : true Load : 10.31 MB Generation No : 1299502115 Uptime (seconds) : 2172 Heap Memory (MB) : 733,82 / 1196,81 C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com tpstats Starting NodeTool Pool Name Active Pending Completed ReadStage 0 0 40646 RequestResponseStage 0 0 30 MutationStage 32 103310 90526 GossipStage 0 0 0 AntiEntropyStage 0 0 0 MigrationStage 0 0 1 MemtablePostFlusher 0 0 69 StreamStage 0 0 0 FlushWriter 0 0 68 FILEUTILS-DELETE-POOL 0 0 42 MiscStage 0 0 0 FlushSorter 0 0 0 InternalResponseStage 0 0 0 HintedHandoff 0 0 0
mutate.sh
Description: application/shellscript
