Hi,
We are testing Cassandra here, we would like to use it to store some data :
- about 1000 inserts / seconds in a CF "RAW" :
Column : TimeUUID (timeuuid of the insert, so 1000 new columns / second)
Row : YYYYMMDDHH of the insert (to minimize the size of rows, the biggest
one is 2GB data), for example for today at 1:30pm, the row of an insert will be
2010070713
- 4 CF we are using for indexes, all have the same structure
Column : TimeUUID of the insert (about 300 new columns per second, each
correspond to the same timeuuid of the CF "RAW".
Row : a String (Customer Id).
The goal is for a Customer, to be able to get all its "RAW" for a time range.
Currently Cassandra is running on 4 nodes / 16 GB RAM each, 8 GB JVM_MAX_MEM
It runs correctly during several days. Last night, we started to have timeout
exception on insert and high cpu load on all nodes.
We stopped inserts. But the CPU remains high (without any insert or read).
We tried to stop and start nodes again, cpu remains high (between 100% and
150%).
So currently, we can't add more data (timeout exceptions, due to high cpu usage
I guess).
Could you help me to understand what's wrong ? :)
Cheers,
Olivier
Here are some data :
We are using Cassandra 0.6.2
Address Status Load Range
Ring
170141183460469231731687303715884105728
192.168.0.1 Up 23.43 GB 42535295865117307932921825928971026432
|<--|
192.168.0.2 Up 23.12 GB 85070591730234615865843651857942052864
| |
192.168.0.3 Up 24.47 GB 127605887595351923798765477786913079296
| |
192.168.0.4 Up 24.14 GB 170141183460469231731687303715884105728
|-->|
Keyspace: system
Read Count: 8
Read Latency: 60.38625 ms.
Write Count: 13
Write Latency: 0.3583076923076923 ms.
Pending Tasks: 0
Column Family: HintsColumnFamily
SSTable count: 1
Space used (live): 103812
Space used (total): 103812
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 5
Read Latency: 83,440 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 1
Key cache size: 1
Key cache hit rate: 0.6666666666666666
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0
Column Family: LocationInfo
SSTable count: 3
Space used (live): 3999
Space used (total): 3999
Memtable Columns Count: 7
Memtable Data Size: 227
Memtable Switch Count: 2
Read Count: 3
Read Latency: 21,964 ms.
Write Count: 13
Write Latency: 0,358 ms.
Pending Tasks: 0
Key cache capacity: 3
Key cache size: 3
Key cache hit rate: 0.2
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0
----------------
Keyspace: MyKeySpace
Read Count: 5
Read Latency: 22737.799 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Column Family: INDEX1
SSTable count: 5
Space used (live): 1502023165
Space used (total): 1502023165
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 1
Read Latency: 0,876 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0
Column Family: INDEX2
SSTable count: 3
Space used (live): 1262977087
Space used (total): 1262977087
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 1
Read Latency: 0,547 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0
Column Family: RAW
SSTable count: 6
Space used (live): 18867913004
Space used (total): 18867913004
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 1
Read Latency: 113685,183 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 3
Key cache hit rate: 0.0
Row cache: disabled
Compacted row minimum size: 215734
Compacted row maximum size: 1055255881
Compacted row mean size: 491170239
Column Family: INDEX3
SSTable count: 6
Space used (live): 1667728231
Space used (total): 1667728231
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 1
Read Latency: 1,241 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0
Column Family: INDEX4
SSTable count: 7
Space used (live): 2620328669
Space used (total): 2620328669
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 1
Read Latency: 1,148 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 0
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0
