Hi, replies to your answers: The CFs that seems to flush more often (users and messages) do not have multiget queries...
There are 4 secondary indexes for users and 3 for messages. I've seen something odd here: the index of the CFs is also stored in a column AND is also indexed. There isn't any query by that secondary index. I should probably erase the index, shouldn't I? I post some lines of the log before 21:35: Node 150: INFO [HintedHandoff:1] 2013-01-30 21:07:43,795 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:07:43,796 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:17:43,796 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:17:43,796 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:27:43,796 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:27:43,797 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 Node 151: INFO [HintedHandoff:1] 2013-01-30 21:04:45,821 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:04:45,822 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [ScheduledTasks:1] 2013-01-30 21:14:30,112 GCInspector.java (line 122) GC for ParNew: 213 ms for 1 collections, 1920192480 used; max is 3886022656 INFO [HintedHandoff:1] 2013-01-30 21:14:45,821 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:14:45,822 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:24:45,822 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:24:45,822 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [ScheduledTasks:1] 2013-01-30 21:28:25,294 GCInspector.java (line 122) GC for ParNew: 236 ms for 1 collections, 2264010544 used; max is 3886022656 INFO [HintedHandoff:1] 2013-01-30 21:34:45,822 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:34:45,822 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 Node 152: INFO [HintedHandoff:1] 2013-01-30 21:02:03,706 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:02:03,707 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 WARN [MemoryMeter:1] 2013-01-30 21:09:57,220 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.15873949606408785 INFO [MemoryMeter:1] 2013-01-30 21:09:57,221 Memtable.java (line 213) CFS(Keyspace='SB_NNCD', ColumnFamily='CF_profilePics') liveRatio is 1.0 (just-counted was 1.0). calculation took 3ms for 301 columns INFO [MemoryMeter:1] 2013-01-30 21:09:57,223 Memtable.java (line 213) CFS(Keyspace='SB_NNCD', ColumnFamily='CF_profilePics.CF_profilePics_pPicMD5_idx') liveRatio is 18.952013237727524 (just-counted was 18.952013237727524). calculation took 2ms for 74 columns INFO [MemoryMeter:1] 2013-01-30 21:09:57,242 Memtable.java (line 213) CFS(Keyspace='SB_NNCD', ColumnFamily='CF_profilePics.CF_profilePics_userSBCode_idx') liveRatio is 12.820966340897906 (just-counted was 12.209597352454495). calculation took 19ms for 74 columns INFO [HintedHandoff:1] 2013-01-30 21:12:03,707 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:12:03,707 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [MemoryMeter:1] 2013-01-30 21:12:47,775 Memtable.java (line 213) CFS(Keyspace='SB_NNCD', ColumnFamily='CF_NewsFeedTimeline') liveRatio is 3.605170679607766 (just-counted was 3.4883667978539283). calculation took 110ms for 1462 columns INFO [HintedHandoff:1] 2013-01-30 21:22:03,707 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:22:03,708 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [MemoryMeter:1] 2013-01-30 21:24:14,795 Memtable.java (line 213) CFS(Keyspace='SB_ACD', ColumnFamily='CF_GSMembers') liveRatio is 5.785411930027831 (just-counted was 5.622629145582293). calculation took 24ms for 463 columns WARN [ScheduledTasks:1] 2013-01-30 21:28:12,744 GCInspector.java (line 145) Heap is 0.7609735541387436 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically WARN [ScheduledTasks:1] 2013-01-30 21:28:12,744 StorageService.java (line 2880) Flushing CFS(Keyspace='SB_ACD', ColumnFamily='CF_users') to relieve memory pressure INFO [ScheduledTasks:1] 2013-01-30 21:28:12,745 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-CF_users.CF_users_userEmail_idx@1131307878(146/182 serialized/live bytes, 4 ops) INFO [ScheduledTasks:1] 2013-01-30 21:28:12,755 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-CF_users.CF_users_userOwnPhone_idx@973751772(146/182 serialized/live bytes, 4 ops) INFO [ScheduledTasks:1] 2013-01-30 21:28:12,755 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-CF_users.CF_users_userOwnPhoneKey_idx@2020641864(146/182 serialized/live bytes, 4 ops) INFO [ScheduledTasks:1] 2013-01-30 21:28:12,756 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-CF_users.CF_users_userSBCode_idx@621414590(146/182 serialized/live bytes, 4 ops) INFO [ScheduledTasks:1] 2013-01-30 21:28:12,756 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-CF_users@902038769(9358482/45802966 serialized/live bytes, 405140 ops) INFO [FlushWriter:11] 2013-01-30 21:28:12,757 Memtable.java (line 264) Writing Memtable-CF_users.CF_users_userEmail_idx@1131307878(146/182 serialized/live bytes, 4 ops) INFO [FlushWriter:11] 2013-01-30 21:28:12,768 Memtable.java (line 305) Completed flushing /raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userEmail_idx-hf-2231-Data.db (234 bytes) for commitlog position ReplayPosition(segmentId=1359573115280, position=6624766) INFO [FlushWriter:11] 2013-01-30 21:28:12,781 Memtable.java (line 264) Writing Memtable-CF_users.CF_users_userOwnPhone_idx@973751772(146/182 serialized/live bytes, 4 ops) INFO [FlushWriter:11] 2013-01-30 21:28:12,793 Memtable.java (line 305) Completed flushing /raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhone_idx-hf-2217-Data.db (265 bytes) for commitlog position ReplayPosition(segmentId=1359573115280, position=6624766) INFO [FlushWriter:11] 2013-01-30 21:28:12,802 Memtable.java (line 264) Writing Memtable-CF_users.CF_users_userOwnPhoneKey_idx@2020641864(146/182 serialized/live bytes, 4 ops) INFO [CompactionExecutor:42] 2013-01-30 21:28:12,813 CompactionTask.java (line 109) Compacting [SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhone_idx-hf-2217-Data.db'), SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhone_idx-hf-2215-Data.db'), SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhone_idx-hf-2216-Data.db'), SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhone_idx-hf-2214-Data.db')] INFO [FlushWriter:11] 2013-01-30 21:28:12,814 Memtable.java (line 305) Completed flushing /raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhoneKey_idx-hf-2209-Data.db (268 bytes) for commitlog position ReplayPosition(segmentId=1359573115280, position=6624766) INFO [FlushWriter:11] 2013-01-30 21:28:12,824 Memtable.java (line 264) Writing Memtable-CF_users.CF_users_userSBCode_idx@621414590(146/182 serialized/live bytes, 4 ops) INFO [CompactionExecutor:43] 2013-01-30 21:28:12,826 CompactionTask.java (line 109) Compacting [SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhoneKey_idx-hf-2206-Data.db'), SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhoneKey_idx-hf-2208-Data.db'), SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhoneKey_idx-hf-2209-Data.db'), SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhoneKey_idx-hf-2207-Data.db')] INFO [FlushWriter:11] 2013-01-30 21:28:12,835 Memtable.java (line 305) Completed flushing /raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userSBCode_idx-hf-2205-Data.db (263 bytes) for commitlog position ReplayPosition(segmentId=1359573115280, position=6624766) INFO [FlushWriter:11] 2013-01-30 21:28:12,838 Memtable.java (line 264) Writing Memtable-CF_users@902038769(9358482/45802966 serialized/live bytes, 405140 ops) INFO [FlushWriter:11] 2013-01-30 21:28:13,513 Memtable.java (line 305) Completed flushing /raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users-hf-3989-Data.db (3358487 bytes) for commitlog position ReplayPosition(segmentId=1359573115280, position=6624766) INFO [CompactionExecutor:42] 2013-01-30 21:28:28,209 CompactionTask.java (line 221) Compacted to [/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhone_idx-hf-2218-Data.db,]. 12,758,536 to 12,758,266 (~99% of original) bytes for 298,733 keys at 0.790337MB/s. Time: 15,395ms. INFO [CompactionExecutor:43] 2013-01-30 21:28:28,258 CompactionTask.java (line 221) Compacted to [/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhoneKey_idx-hf-2210-Data.db,]. 12,827,751 to 12,827,331 (~99% of original) bytes for 298,767 keys at 0.792812MB/s. Time: 15,430ms. INFO [MemoryMeter:1] 2013-01-30 21:31:30,638 Memtable.java (line 213) CFS(Keyspace='SB_NNCD', ColumnFamily='CF_userPhoneAgenda') liveRatio is 9.966352958143455 (just-counted was 6.201877882350339). calculation took 24ms for 6574 columns INFO [HintedHandoff:1] 2013-01-30 21:32:03,707 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:32:03,708 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 WARN [ScheduledTasks:1] 2013-01-30 21:33:41,079 GCInspector.java (line 145) Heap is 0.7632044026868381 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically WARN [ScheduledTasks:1] 2013-01-30 21:33:41,079 StorageService.java (line 2880) Flushing CFS(Keyspace='SB_ACD', ColumnFamily='CF_R2Notifications') to relieve memory pressure INFO [ScheduledTasks:1] 2013-01-30 21:33:41,080 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-CF_R2Notifications@1187465569(760532/33771575 serialized/live bytes, 93136 ops) INFO [FlushWriter:12] 2013-01-30 21:33:41,105 Memtable.java (line 264) Writing Memtable-CF_R2Notifications@1187465569(760532/33771575 serialized/live bytes, 93136 ops) INFO [FlushWriter:12] 2013-01-30 21:33:42,802 Memtable.java (line 305) Completed flushing /raid0/cassandra/data/SB_ACD/CF_R2Notifications/SB_ACD-CF_R2Notifications-hf-4175-Data.db (639626 bytes) for commitlog position ReplayPosition(segmentId=1359573115280, position=11032499) Node 153: INFO [HintedHandoff:1] 2013-01-30 21:01:45,310 HintedHandOffManager.java (line 296) Started hinted handoff for token: 56713727820156407428984779325531226112 with IP: /10.0.0.152 INFO [HintedHandoff:1] 2013-01-30 21:01:45,311 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.152 INFO [MemoryMeter:1] 2013-01-30 21:05:45,297 Memtable.java (line 213) CFS(Keyspace='SB_ACD', ColumnFamily='CF_users') liveRatio is 3.9183034832767603 (just-counted was 3.9183034832767603). calculation took 255ms for 58995 columns INFO [MemoryMeter:1] 2013-01-30 21:07:24,736 Memtable.java (line 213) CFS(Keyspace='SB_NNCD', ColumnFamily='CF_NewsFeedTimeline') liveRatio is 3.6493489644964034 (just-counted was 3.471672980060775). calculation took 107ms for 1395 columns INFO [ScheduledTasks:1] 2013-01-30 21:07:41,673 GCInspector.java (line 122) GC for ParNew: 215 ms for 1 collections, 2310767440 used; max is 3886022656 INFO [HintedHandoff:1] 2013-01-30 21:11:45,316 HintedHandOffManager.java (line 296) Started hinted handoff for token: 56713727820156407428984779325531226112 with IP: /10.0.0.152 INFO [HintedHandoff:1] 2013-01-30 21:11:45,318 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.152 INFO [MemoryMeter:1] 2013-01-30 21:14:40,052 Memtable.java (line 213) CFS(Keyspace='SB_NNCD', ColumnFamily='CF_profilePics.CF_profilePics_pPicMD5_idx') liveRatio is 18.87210642710637 (just-counted was 18.854822335025382). calculation took 3ms for 80 columns INFO [MemoryMeter:1] 2013-01-30 21:14:40,054 Memtable.java (line 213) CFS(Keyspace='SB_NNCD', ColumnFamily='CF_profilePics.CF_profilePics_userSBCode_idx') liveRatio is 12.252182911928061 (just-counted was 11.273365071410675). calculation took 2ms for 81 columns WARN [MemoryMeter:1] 2013-01-30 21:14:40,072 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.12200431775304404 INFO [MemoryMeter:1] 2013-01-30 21:14:40,072 Memtable.java (line 213) CFS(Keyspace='SB_NNCD', ColumnFamily='CF_profilePics') liveRatio is 1.0 (just-counted was 1.0). calculation took 3ms for 315 columns INFO [ScheduledTasks:1] 2013-01-30 21:14:48,898 GCInspector.java (line 122) GC for ParNew: 201 ms for 1 collections, 2512549552 used; max is 3886022656 INFO [MemoryMeter:1] 2013-01-30 21:15:32,061 Memtable.java (line 213) CFS(Keyspace='SB_NNCD', ColumnFamily='CF_SBCodes') liveRatio is 32.58987080513604 (just-counted was 31.082180634662326). calculation took 2ms for 52 columns INFO [ScheduledTasks:1] 2013-01-30 21:21:08,322 GCInspector.java (line 122) GC for ParNew: 239 ms for 1 collections, 2576069224 used; max is 3886022656 INFO [HintedHandoff:1] 2013-01-30 21:21:45,311 HintedHandOffManager.java (line 296) Started hinted handoff for token: 56713727820156407428984779325531226112 with IP: /10.0.0.152 INFO [HintedHandoff:1] 2013-01-30 21:21:45,312 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.152 INFO [MemoryMeter:1] 2013-01-30 21:23:46,635 Memtable.java (line 213) CFS(Keyspace='SB_ACD', ColumnFamily='CF_GSData') liveRatio is 7.342488214798115 (just-counted was 7.342488214798115). calculation took 0ms for 116 columns INFO [MemoryMeter:1] 2013-01-30 21:24:14,793 Memtable.java (line 213) CFS(Keyspace='SB_ACD', ColumnFamily='CF_GSMembers') liveRatio is 5.753869723542124 (just-counted was 5.630372102807786). calculation took 24ms for 456 columns INFO [ScheduledTasks:1] 2013-01-30 21:30:18,511 GCInspector.java (line 122) GC for ParNew: 226 ms for 1 collections, 2242332984 used; max is 3886022656 INFO [MemoryMeter:1] 2013-01-30 21:30:46,426 Memtable.java (line 213) CFS(Keyspace='SB_ACD', ColumnFamily='CF_R1Notifications') liveRatio is 6.832365653565714 (just-counted was 6.675185638944877). calculation took 43ms for 860 columns INFO [HintedHandoff:1] 2013-01-30 21:31:45,311 HintedHandOffManager.java (line 296) Started hinted handoff for token: 56713727820156407428984779325531226112 with IP: /10.0.0.152 INFO [HintedHandoff:1] 2013-01-30 21:31:45,313 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.152 INFO [ScheduledTasks:1] 2013-01-30 21:33:12,105 GCInspector.java (line 122) GC for ParNew: 261 ms for 1 collections, 2603400728 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:33:21,312 GCInspector.java (line 122) GC for ParNew: 248 ms for 1 collections, 3191815000 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:33:23,315 GCInspector.java (line 122) GC for ParNew: 201 ms for 1 collections, 3471315456 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:33:35,323 GCInspector.java (line 122) GC for ParNew: 204 ms for 1 collections, 1636739048 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:33:51,462 GCInspector.java (line 122) GC for ParNew: 239 ms for 1 collections, 2789318016 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:34:00,805 GCInspector.java (line 122) GC for ParNew: 217 ms for 1 collections, 2959924968 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:34:05,008 GCInspector.java (line 122) GC for ParNew: 217 ms for 1 collections, 2900952792 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:34:17,262 GCInspector.java (line 122) GC for ParNew: 231 ms for 1 collections, 2368279416 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:34:19,267 GCInspector.java (line 122) GC for ParNew: 201 ms for 1 collections, 2297887816 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:34:28,326 GCInspector.java (line 122) GC for ParNew: 215 ms for 1 collections, 2892129512 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:34:30,328 GCInspector.java (line 122) GC for ParNew: 266 ms for 1 collections, 3168051328 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:34:42,552 GCInspector.java (line 122) GC for ParNew: 249 ms for 1 collections, 2330298920 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:34:46,554 GCInspector.java (line 122) GC for ParNew: 243 ms for 1 collections, 2496880592 used; max is 3886022656 WARN [ScheduledTasks:1] 2013-01-30 21:35:02,994 GCInspector.java (line 145) Heap is 0.7641879728660028 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically WARN [ScheduledTasks:1] 2013-01-30 21:35:02,995 StorageService.java (line 2880) Flushing CFS(Keyspace='SB_ACD', ColumnFamily='CF_R2Notifications') to relieve memory pressure INFO [ScheduledTasks:1] 2013-01-30 21:35:02,995 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-CF_R2Notifications@1630759928(1137418/88800160 serialized/live bytes, 171909 ops) INFO [FlushWriter:4] 2013-01-30 21:35:03,016 Memtable.java (line 264) Writing Memtable-CF_R2Notifications@1630759928(1137418/88800160 serialized/live bytes, 171909 ops) INFO [ScheduledTasks:1] 2013-01-30 21:35:04,293 GCInspector.java (line 122) GC for ParNew: 455 ms for 2 collections, 2926515240 used; max is 3886022656 INFO [FlushWriter:4] 2013-01-30 21:35:05,798 Memtable.java (line 305) Completed flushing /raid0/cassandra/data/SB_ACD/CF_R2Notifications/SB_ACD-CF_R2Notifications-hf-3832-Data.db (790256 bytes) for commitlog position ReplayPosition(segmentId=1359573697867, position=11748063) INFO [ScheduledTasks:1] 2013-01-30 21:35:06,344 GCInspector.java (line 122) GC for ParNew: 204 ms for 1 collections, 3227837984 used; max is 3886022656 INFO [MemoryMeter:1] 2013-01-30 21:35:06,369 Memtable.java (line 213) CFS(Keyspace='SB_ACD', ColumnFamily='CF_R2Notifications') liveRatio is 37.62080440650366 (just-counted was 12.784246424797216). calculation took 7181ms for 2683 columns INFO [ScheduledTasks:1] 2013-01-30 21:35:08,345 GCInspector.java (line 122) GC for ParNew: 216 ms for 1 collections, 3586092576 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:35:15,625 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 1130 ms for 2 collections, 602554696 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:35:22,887 GCInspector.java (line 122) GC for ParNew: 253 ms for 1 collections, 1362776544 used; max is 3886022656 INFO [ScheduledTasks:1] 2013-01-30 21:35:23,888 GCInspector.java (line 122) GC for ParNew: 313 ms for 1 collections, 1473067952 used; max is 3886022656 Node 154: INFO [HintedHandoff:1] 2013-01-30 21:05:00,728 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:05:00,728 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:15:00,728 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:15:00,729 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [ScheduledTasks:1] 2013-01-30 21:21:28,248 GCInspector.java (line 122) GC for ParNew: 356 ms for 1 collections, 1272387016 used; max is 3886022656 INFO [HintedHandoff:1] 2013-01-30 21:25:00,728 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:25:00,729 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:35:00,728 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:35:00,729 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 Node 155: INFO [HintedHandoff:1] 2013-01-30 21:05:07,916 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:05:07,917 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:15:07,916 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:15:07,917 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:25:07,916 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:25:07,918 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:35:07,917 HintedHandOffManager.java (line 296) Started hinted handoff for token: 85070591730234615865843651857942052864 with IP: /10.0.0.153 INFO [HintedHandoff:1] 2013-01-30 21:35:07,918 HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 I've read that the message 'Finished hinted handoff of 0 rows to endpoint ...' is because a tombstone not being deleted, and that I shouldn't worry about it. I'm also talking with Amazon to check if there is a problem with their service, the nodes 152 and 153 where using more cpu than the other four nodes (but they have been like this all over the week, not just a peak at the very same our for the last 3 days) All the Cassandra configuration is at default values (the heap size, row caches disabled, etc.) Thanks for your support, 2013/1/31 Alain RODRIGUEZ <arodr...@gmail.com>: > @Bryan > > "Other space is used but can be more easily controlled by tuning for > - memtable > - compaction > - key cache > - row cache" > > Isn't row cache stored off-heap since a while ? > > > 2013/1/31 Bryan Talbot <btal...@aeriagames.com> >> >> On Wed, Jan 30, 2013 at 2:44 PM, Guillermo Barbero >> <guillermo.barb...@spotbros.com> wrote: >>> >>> WARN [MemoryMeter:1] 2013-01-30 21:37:48,079 Memtable.java (line 202) >>> setting live ratio to maximum of 64.0 instead of 751.6512549537648 >>> >> >> This looks interesting. Doesn't this mean that the ratio of space used >> for that CF for memory to serialized form is 751:1 but was forced to a lower >> "sane" value? >> >> -Bryan >> > -- Guillermo Barbero - Backend Team Spotbros Technologies
