Hi guys,
We're currently testing an application against a very high load, which
runs against Cassandra 0.6.13 (I know, we just never got the time to
upgrade).
The nature of our app is that it will write to two different
SuperColumnFamilies in bursts, and to some other columnfamilies less
frequently.
About every 10 mins a very large number of subcolumns will be written
to those superColumnFamilies (> 1 million). The writes are mainly
overwrites, there are no deletes.
when I do a cfstats on the SuperColumnFamilies I see something like this:
SSTable count: 4
Space used (live): 1611778835
Space used (total): 1611778835
Memtable Columns Count: 82844
Memtable Data Size: 4439246
Memtable Switch Count: 5262
Read Count: 4633810
Read Latency: 3.727 ms.
Write Count: 29739347
Write Latency: 0.028 ms.
Pending Tasks: 0
Which tells me the Write Latency is not extremely high, but I can see
in my logs that some batch writes, which contain about 1000 entries,
take over 2 seconds to execute.
I can see that during the times the writing gets slow there are ~3000
pending tasks, but they disappear quickly.
Is there anything I can do to tune Cassandra so the writes won't take
that long? Maybe increasing the MemtableOperationsInMillions so the
memtables get flushed less often?
Or is the load just too high and should I solve this by adding more
nodes? We are currently testing with 3, replication factor 2.
I can also see that Cassandra gradually takes more and more memory,
eventually filling up the 16GB that is assigned to it, although it
doesn't go out of memory.
Is this normal behaviour? I expected to see more of a sawtooth...
All help is appreciated.
Thanks,
Stefan
