Your latencies and distribution look fine. How big/what types of queries are you issuing? Are you issuing a lot of large multigets?
Also, do either of these column families have secondary indexes? On Wed, Jan 30, 2013 at 2:59 PM, Guillermo Barbero <[email protected]> wrote: > Iep, > > I missed the attachment... > > Also, these are the cfstats of 152: > > Column Family: CF_SBMessages > SSTable count: 3 > Space used (live): 967238560 > Space used (total): 967238560 > Number of Keys (estimate): 3263232 > Memtable Columns Count: 2112 > Memtable Data Size: 577135 > Memtable Switch Count: 5 > Read Count: 20702 > Read Latency: 0.181 ms. > Write Count: 19888 > Write Latency: 0.059 ms. > Pending Tasks: 0 > Bloom Filter False Postives: 14 > Bloom Filter False Ratio: 0.09375 > Bloom Filter Space Used: 6171168 > Compacted row minimum size: 536 > Compacted row maximum size: 24601 > Compacted row mean size: 850 > > Column Family: CF_users > SSTable count: 3 > Space used (live): 152205376 > Space used (total): 152205376 > Number of Keys (estimate): 343040 > Memtable Columns Count: 154398 > Memtable Data Size: 21159410 > Memtable Switch Count: 9 > Read Count: 11816 > Read Latency: 1.348 ms. > Write Count: 128751 > Write Latency: 0.090 ms. > Pending Tasks: 0 > Bloom Filter False Postives: 0 > Bloom Filter False Ratio: 0.00000 > Bloom Filter Space Used: 688464 > Compacted row minimum size: 61 > Compacted row maximum size: 3311 > Compacted row mean size: 1235 > > And the cfstats of 153: > > Column Family: CF_SBMessages > SSTable count: 5 > Space used (live): 965495648 > Space used (total): 965495648 > Number of Keys (estimate): 3257216 > Memtable Columns Count: 56541 > Memtable Data Size: 15699960 > Memtable Switch Count: 2 > Read Count: 22475 > Read Latency: 0.142 ms. > Write Count: 21719 > Write Latency: 0.073 ms. > Pending Tasks: 0 > Bloom Filter False Postives: 43 > Bloom Filter False Ratio: 0.19545 > Bloom Filter Space Used: 6161032 > Compacted row minimum size: 536 > Compacted row maximum size: 11864 > Compacted row mean size: 872 > > Column Family: CF_users > SSTable count: 2 > Space used (live): 148762893 > Space used (total): 148762893 > Number of Keys (estimate): 333312 > Memtable Columns Count: 129725 > Memtable Data Size: 17144125 > Memtable Switch Count: 4 > Read Count: 7440 > Read Latency: 1.329 ms. > Write Count: 127465 > Write Latency: 0.093 ms. > Pending Tasks: 0 > Bloom Filter False Postives: 0 > Bloom Filter False Ratio: 0.00000 > Bloom Filter Space Used: 694112 > Compacted row minimum size: 61 > Compacted row maximum size: 3311 > Compacted row mean size: 1298 > > The messages are in a keyspace with a replication factor of 2 and the > users are in a keyspace with a replication factor of 3. > > Ah, and we use the RandomPartitioner. > > Thanks again > > > 2013/1/30 Guillermo Barbero <[email protected]> >> >> Guys, >> >> First thanks a lot for your answers.. now I have a little more info, lets >> see if this helps: >> >> As I said before we are using Cassandra 1.1.7 on a 6 nodes(Ips: 150-155) >> (Amazon EC2 XL instances 17GB RAM) and the problem is that in the last weeks >> we are seing that the performace of the cluster falls.. PHP commands that >> usually lasted a few milliseconds last up to 15 seconds for just 2/3 minutes >> every now and then (no specific pattern) (see commands timing in attached >> picture). >> >> Today we have found this: >> >> Prior to a fall in performance on all Cassandra nodes log I can see this: >> >> Node 150: >> >> INFO [GossipTasks:1] 2013-01-30 21:35:23,514 Gossiper.java (line 831) >> InetAddress /10.0.0.152 is now dead. >> INFO [GossipStage:1] 2013-01-30 21:35:40,666 Gossiper.java (line 817) >> InetAddress /10.0.0.152 is now UP >> INFO [HintedHandoff:1] 2013-01-30 21:35:40,667 HintedHandOffManager.java >> (line 296) Started hinted handoff for token: >> 56713727820156407428984779325531226112 with IP: /10.0.0.152 >> INFO [HintedHandoff:1] 2013-01-30 21:35:41,266 ColumnFamilyStore.java (line >> 659) Enqueuing flush of Memtable-HintsColumnFamily@1264710747(84317/476272 >> serialized/live bytes, 298 ops) >> INFO [FlushWriter:2879] 2013-01-30 21:35:41,267 Memtable.java (line 264) >> Writing Memtable-HintsColumnFamily@1264710747(84317/476272 serialized/live >> bytes, 298 ops) >> INFO [FlushWriter:2879] 2013-01-30 21:35:41,282 Memtable.java (line 305) >> Completed flushing >> /raid0/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-276-Data.db >> (84559 bytes) for commitlog position >> ReplayPosition(segmentId=1355151995930, position=32659991) >> INFO [CompactionExecutor:9098] 2013-01-30 21:35:41,283 CompactionTask.java >> (line 109) Compacting >> [SSTableReader(path='/raid0/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-275-Data.db'), >> >> SSTableReader(path='/raid0/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-276-Data.db')] >> >> Node 151: >> >> INFO [GossipTasks:1] 2013-01-30 21:35:25,689 Gossiper.java (line 831) >> InetAddress /10.0.0.152 is now dead. >> INFO [GossipStage:1] 2013-01-30 21:35:40,677 Gossiper.java (line 817) >> InetAddress /10.0.0.152 is now UP >> INFO [HintedHandoff:1] 2013-01-30 21:35:40,677 HintedHandOffManager.java >> (line 296) Started hinted handoff for token: >> 56713727820156407428984779325531226112 with IP: /10.0.0.152 >> INFO [HintedHandoff:1] 2013-01-30 21:35:41,548 ColumnFamilyStore.java (line >> 659) Enqueuing flush of Memtable-HintsColumnFamily@1453632632(93461/191418 >> serialized/live bytes, 316 ops) >> INFO [FlushWriter:2636] 2013-01-30 21:35:41,549 Memtable.java (line 264) >> Writing Memtable-HintsColumnFamily@1453632632(93461/191418 serialized/live >> bytes, 316 ops) >> INFO [FlushWriter:2636] 2013-01-30 21:35:41,563 Memtable.java (line 305) >> Completed flushing >> /raid0/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-266-Data.db >> (93711 bytes) for commitlog position >> ReplayPosition(segmentId=1354832000013, position=20033362) >> >> Node 153: >> >> INFO [GossipTasks:1] 2013-01-30 21:35:24,822 Gossiper.java (line 831) >> InetAddress /10.0.0.152 is now dead. >> INFO [ScheduledTasks:1] 2013-01-30 21:35:24,896 GCInspector.java (line 122) >> GC for ParNew: 294 ms for 1 collections, 1539602448 used; max is 3886022656 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:26,897 GCInspector.java (line 122) >> GC for ParNew: 202 ms for 1 collections, 1829975768 used; max is 3886022656 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:27,898 GCInspector.java (line 122) >> GC for ParNew: 234 ms for 1 collections, 2037767104 used; max is 3886022656 >> INFO [MemoryMeter:1] 2013-01-30 21:35:31,151 Memtable.java (line 213) >> CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is >> 1.4195889775077013 (just-counted was 1.4195889775077013). calculation took >> 109ms for 64 columns >> INFO [MemoryMeter:1] 2013-01-30 21:35:37,806 Memtable.java (line 213) >> CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is >> 1.5276925191259334 (just-counted was 1.5276925191259334). calculation took >> 11ms for 129 columns >> INFO [GossipStage:1] 2013-01-30 21:35:40,735 Gossiper.java (line 817) >> InetAddress /10.0.0.152 is now UP >> INFO [HintedHandoff:1] 2013-01-30 21:35:40,742 HintedHandOffManager.java >> (line 296) Started hinted handoff for token: >> 56713727820156407428984779325531226112 with IP: /10.0.0.152 >> INFO [MemoryMeter:1] 2013-01-30 21:35:43,184 Memtable.java (line 213) >> CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is >> 1.561312033901826 (just-counted was 1.561312033901826). calculation took >> 234ms for 144 columns >> INFO [HintedHandoff:1] 2013-01-30 21:35:43,549 ColumnFamilyStore.java (line >> 659) Enqueuing flush of Memtable-HintsColumnFamily@1120216043(96278/187900 >> serialized/live bytes, 288 ops) >> >> Node 154: >> >> INFO [HintedHandoff:1] 2013-01-30 21:35:00,729 HintedHandOffManager.java >> (line 392) Finished hinted handoff of 0 rows to endpoint /10.0.0.153 >> INFO [GossipTasks:1] 2013-01-30 21:35:26,432 Gossiper.java (line 831) >> InetAddress /10.0.0.152 is now dead. >> INFO [GossipStage:1] 2013-01-30 21:35:40,669 Gossiper.java (line 817) >> InetAddress /10.0.0.152 is now UP >> INFO [HintedHandoff:1] 2013-01-30 21:35:40,669 HintedHandOffManager.java >> (line 296) Started hinted handoff for token: >> 56713727820156407428984779325531226112 with IP: /10.0.0.152 >> INFO [HintedHandoff:1] 2013-01-30 21:35:41,208 ColumnFamilyStore.java (line >> 659) Enqueuing flush of Memtable-HintsColumnFamily@908822645(92404/185145 >> serialized/live bytes, 286 ops) >> INFO [FlushWriter:2496] 2013-01-30 21:35:41,209 Memtable.java (line 264) >> Writing Memtable-HintsColumnFamily@908822645(92404/185145 serialized/live >> bytes, 286 ops) >> >> >> Node 155: >> >> INFO [GossipTasks:1] 2013-01-30 21:35:27,117 Gossiper.java (line 831) >> InetAddress /10.0.0.152 is now dead. >> INFO [GossipStage:1] 2013-01-30 21:35:40,668 Gossiper.java (line 817) >> InetAddress /10.0.0.152 is now UP >> INFO [HintedHandoff:1] 2013-01-30 21:35:40,669 HintedHandOffManager.java >> (line 296) Started hinted handoff for token: >> 56713727820156407428984779325531226112 with IP: /10.0.0.152 >> INFO [HintedHandoff:1] 2013-01-30 21:35:41,279 ColumnFamilyStore.java (line >> 659) Enqueuing flush of Memtable-HintsColumnFamily@1083199446(94578/190654 >> serialized/live bytes, 316 ops) >> >> Meanwhile on node 152: >> >> INFO [FlushWriter:12] 2013-01-30 21:33:42,802 Memtable.java (line 305) >> Completed flushing >> /raid0/cassandra/data/SB_ACD/CF_R2Notifications/SB_ACD-CF_R2Notifications-hf-4175-Data.db >> (639626 bytes) for commitlog position >> ReplayPosition(segmentId=1359573115280, position=11032499) >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,701 GCInspector.java (line 122) >> GC for ParNew: 35087 ms for 3 collections, 442940272 used; max is 3886022656 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,934 StatusLogger.java (line 57) >> Pool Name Active Pending Blocked >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,941 StatusLogger.java (line 72) >> ReadStage 32 343 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,942 StatusLogger.java (line 72) >> RequestResponseStage 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,942 StatusLogger.java (line 72) >> ReadRepairStage 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,942 StatusLogger.java (line 72) >> MutationStage 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,943 StatusLogger.java (line 72) >> ReplicateOnWriteStage 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,943 StatusLogger.java (line 72) >> GossipStage 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,943 StatusLogger.java (line 72) >> AntiEntropyStage 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,944 StatusLogger.java (line 72) >> MigrationStage 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,944 StatusLogger.java (line 72) >> StreamStage 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,945 StatusLogger.java (line 72) >> MemtablePostFlusher 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,945 StatusLogger.java (line 72) >> FlushWriter 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,945 StatusLogger.java (line 72) >> MiscStage 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,946 StatusLogger.java (line 72) >> commitlog_archiver 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,946 StatusLogger.java (line 72) >> InternalResponseStage 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,946 StatusLogger.java (line 72) >> HintedHandoff 0 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,947 StatusLogger.java (line 77) >> CompactionManager 0 0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,947 StatusLogger.java (line 89) >> MessagingService n/a 0,24 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,947 StatusLogger.java (line 99) >> Cache Type Size Capacity >> KeysToSave Provider >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,948 StatusLogger.java (line >> 100) KeyCache 133569 2184533 >> all >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,948 StatusLogger.java (line >> 106) RowCache 0 0 >> all >> org.apache.cassandra.cache.SerializingCacheProvider >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,948 StatusLogger.java (line >> 113) ColumnFamily Memtable ops,data >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,948 StatusLogger.java (line >> 116) system.NodeIdInfo 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,949 StatusLogger.java (line >> 116) system.IndexInfo 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,949 StatusLogger.java (line >> 116) system.LocationInfo 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,949 StatusLogger.java (line >> 116) system.Versions 3,103 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,949 StatusLogger.java (line >> 116) system.schema_keyspaces 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:40,950 StatusLogger.java (line >> 116) system.Migrations 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,059 StatusLogger.java (line >> 116) system.schema_columnfamilies 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,060 StatusLogger.java (line >> 116) system.schema_columns 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,060 StatusLogger.java (line >> 116) system.HintsColumnFamily 1,2369 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,060 StatusLogger.java (line >> 116) system.Schema 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,080 StatusLogger.java (line >> 116) SB_NNCD.CF_mailPipes 9241,30380203 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,080 StatusLogger.java (line >> 116) SB_NNCD.CF_profilePics 427,1173642 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,080 StatusLogger.java (line >> 116) SB_NNCD.CF_SBVars 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,080 StatusLogger.java (line >> 116) SB_NNCD.CF_SBCounters 5,232 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,092 StatusLogger.java (line >> 116) SB_NNCD.CF_userPhoneAgenda 7919,2332898 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,093 StatusLogger.java (line >> 116) SB_NNCD.CF_BSLUsers 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,093 StatusLogger.java (line >> 116) SB_NNCD.CF_mailTimeline 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,093 StatusLogger.java (line >> 116) SB_NNCD.CF_SBMessages 54390,15054201 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,093 StatusLogger.java (line >> 116) SB_NNCD.CF_NewsFeedTimeline 1866,7681861 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,094 StatusLogger.java (line >> 116) SB_NNCD.CF_BSLSpots 1,616 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,094 StatusLogger.java (line >> 116) SB_NNCD.CF_SBUniqueIds 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,094 StatusLogger.java (line >> 116) SB_NNCD.CF_SBChecks 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,094 StatusLogger.java (line >> 116) SB_NNCD.CF_NFTimelines 423,533156 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,094 StatusLogger.java (line >> 116) SB_NNCD.CF_SBAppUsersProfilePic 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,095 StatusLogger.java (line >> 116) SB_NNCD.CF_SBLikes 16,565 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,095 StatusLogger.java (line >> 116) SB_NNCD.CF_SBCodes 154,93735 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,095 StatusLogger.java (line >> 116) SB_NNCD.CF_attachmentsPerUser 80,107551 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,095 StatusLogger.java (line >> 116) SB_NNCD.CF_comments 58,77581 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,096 StatusLogger.java (line >> 116) SB_NNCD.CF_attachments 641,247796 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,096 StatusLogger.java (line >> 116) SB_NNCD.CF_SBStats 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,096 StatusLogger.java (line >> 116) SB_NNCD.CF_userReversedPhoneAgenda 9047,7132129 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,096 StatusLogger.java (line >> 116) SB_ACD.CF_GSMembers 823,1206779 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,113 StatusLogger.java (line >> 116) SB_ACD.CF_NSLoad 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,114 StatusLogger.java (line >> 116) SB_ACD.CF_sessions 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,115 StatusLogger.java (line >> 116) SB_ACD.CF_userGS 193,577229 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,116 StatusLogger.java (line >> 116) SB_ACD.CF_userApps 8,170 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,116 StatusLogger.java (line >> 116) SB_ACD.CF_GSData 126,3503 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,117 StatusLogger.java (line >> 116) SB_ACD.CF_R2Notifications 1906,58481 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,118 StatusLogger.java (line >> 116) SB_ACD.CF_uniqueKeys 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,155 StatusLogger.java (line >> 116) SB_ACD.CF_userContactWannabes 953,438671 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,157 StatusLogger.java (line >> 116) SB_ACD.CF_SBApps 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,158 StatusLogger.java (line >> 116) SB_ACD.CF_SBAppUsers 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,159 StatusLogger.java (line >> 116) SB_ACD.CF_R1Notifications 1354,2070513 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,159 StatusLogger.java (line >> 116) SB_ACD.CF_SBVotes 463,230386 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,161 StatusLogger.java (line >> 116) SB_ACD.CF_userContacts 2968,4849519 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,168 StatusLogger.java (line >> 116) SB_ACD.CF_sessionsSuspended 0,0 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,168 StatusLogger.java (line >> 116) SB_ACD.CF_userBlockedList 5,207 >> INFO [ScheduledTasks:1] 2013-01-30 21:35:41,172 StatusLogger.java (line >> 116) SB_ACD.CF_users 65386,10852776 >> WARN [ScheduledTasks:1] 2013-01-30 21:37:05,895 GCInspector.java (line 145) >> Heap is 0.7514124302624755 full. You may need to reduce memtable and/or >> cache sizes. Cassandra will now flush up to the two largest memtables to >> free up memory. Adjust flush_largest_memtables_at threshold in >> cassandra.yaml if you don't want Cassandra to do this automatically >> WARN [ScheduledTasks:1] 2013-01-30 21:37:05,895 StorageService.java (line >> 2880) Flushing CFS(Keyspace='SB_NNCD', ColumnFamily='CF_mailPipes') to >> relieve memory pressure >> INFO [ScheduledTasks:1] 2013-01-30 21:37:05,896 ColumnFamilyStore.java >> (line 659) Enqueuing flush of >> Memtable-CF_mailPipes@192792072(6868296/30790355 serialized/live bytes, 9380 >> ops) >> INFO [FlushWriter:13] 2013-01-30 21:37:05,909 Memtable.java (line 264) >> Writing Memtable-CF_mailPipes@192792072(6868296/30790355 serialized/live >> bytes, 9380 ops) >> INFO [FlushWriter:13] 2013-01-30 21:37:06,385 Memtable.java (line 305) >> Completed flushing >> /raid0/cassandra/data/SB_NNCD/CF_mailPipes/SB_NNCD-CF_mailPipes-hf-2165-Data.db >> (2049401 bytes) for commitlog position >> ReplayPosition(segmentId=1359573115280, position=13578602) >> WARN [MemoryMeter:1] 2013-01-30 21:37:48,079 Memtable.java (line 202) >> setting live ratio to maximum of 64.0 instead of 751.6512549537648 >> INFO [MemoryMeter:1] 2013-01-30 21:37:48,079 Memtable.java (line 213) >> CFS(Keyspace='SB_ACD', ColumnFamily='CF_R2Notifications') liveRatio is 64.0 >> (just-counted was 64.0). calculation took 61ms for 8 columns >> INFO [MemoryMeter:1] 2013-01-30 21:40:47,274 Memtable.java (line 213) >> CFS(Keyspace='SB_ACD', ColumnFamily='CF_userGS') liveRatio is >> 5.309650177091864 (just-counted was 5.309650177091864). calculation took >> 11ms for 170 columns >> INFO [HintedHandoff:1] 2013-01-30 21:42:03,709 HintedHandOffManager.java >> (line 296) Started hinted handoff for token: >> 85070591730234615865843651857942052864 with IP: /10.0.0.153 >> INFO [HintedHandoff:1] 2013-01-30 21:42:03,712 ColumnFamilyStore.java (line >> 659) Enqueuing flush of Memtable-HintsColumnFamily@1676047624(950/2369 >> serialized/live bytes, 2 ops) >> INFO [FlushWriter:14] 2013-01-30 21:42:03,713 Memtable.java (line 264) >> Writing Memtable-HintsColumnFamily@1676047624(950/2369 serialized/live >> bytes, 2 ops) >> INFO [FlushWriter:14] 2013-01-30 21:42:03,730 Memtable.java (line 305) >> Completed flushing >> /raid0/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-122-Data.db >> (1016 bytes) for commitlog position ReplayPosition(segmentId=1359573115280, >> position=17323736) >> INFO [CompactionExecutor:48] 2013-01-30 21:42:03,731 CompactionTask.java >> (line 109) Compacting >> [SSTableReader(path='/raid0/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-122-Data.db'), >> >> SSTableReader(path='/raid0/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-121-Data.db')] >> >> The cluster is perfectly balanced: >> >> Note: Ownership information does not include topology, please specify a >> keyspace. >> Address DC Rack Status State Load Owns >> Token >> >> 141784319550391032739561396922763706368 >> 10.0.0.150 datacenter1 rack1 Up Normal 13.25 GB >> 16.67% 0 >> 10.0.0.151 datacenter1 rack1 Up Normal 10.31 GB >> 16.67% 28356863910078203714492389662765613056 >> 10.0.0.152 datacenter1 rack1 Up Normal 8.58 GB >> 16.67% 56713727820156407428984779325531226112 >> 10.0.0.153 datacenter1 rack1 Up Normal 8.1 GB >> 16.67% 85070591730234615865843651857942052864 >> 10.0.0.154 datacenter1 rack1 Up Normal 8.28 GB >> 16.67% 113427455640312814857969558651062452224 >> 10.0.0.155 datacenter1 rack1 Up Normal 9.38 GB >> 16.67% 141784319550391032739561396922763706368 >> >> >> I guess something happens on 152 that makes other nodes believe it is dead, >> then does something to fix itself and comes back after (messing up >> performance in the meanwhile) .Does this makes any sense? Any clue of what >> might be going on? I can post more logs if you guys need >> >> Thanks again!! >> >> Guillermo >> >> >> >> 2013/1/30 Nate McCall <[email protected]> >>> >>> What's the output of nodetool cfstats for those 2 column families on >>> cassNode2 and cassNode3? And what is the replication factor for this >>> cluster? >>> >>> Per the previous reply, nodetool ring should show each of your nodes >>> with ~16.7% of the data if well balanced. >>> >>> Also, the auto-detection for memory sizes in the startup script is a >>> little off w/r/t m1.xlarge because of the 'slightly less than 16gb' of >>> ram. It usually ends up allocating 4g/400m (max and young) whereas >>> 8g/800m will give you some more breathing room. >>> >>> On Wed, Jan 30, 2013 at 12:07 PM, Bryan Talbot <[email protected]> >>> wrote: >>> > My guess is that those one or two nodes with the gc pressure also have >>> > more >>> > rows in your big CF. More rows could be due to imbalanced distribution if >>> > your'e not using a random partitioner or from those nodes not yet removing >>> > deleted rows which other nodes may have done. >>> > >>> > JVM heap space is used for a few things which scale with key count >>> > including: >>> > - bloom filter (for C* < 1.2) >>> > - index samples >>> > >>> > Other space is used but can be more easily controlled by tuning for >>> > - memtable >>> > - compaction >>> > - key cache >>> > - row cache >>> > >>> > >>> > So, if those nodes have more rows (check using "nodetool ring" or >>> > "nodetool >>> > cfstats") than the others you can try to: >>> > - reduce the number of rows by adding nodes, run manual / tune compactions >>> > to remove rows with expired tombstones, etc. >>> > - increase bloom filter fp chance >>> > - increase jvm heap size (don't go too big) >>> > - disable key or row cache >>> > - increase index sample interval >>> > >>> > Not all of those things are generally good especially to the extreme so >>> > don't go setting a 20 GB jvm heap without understanding the consequences >>> > for >>> > example. >>> > >>> > -Bryan >>> > >>> > >>> > On Wed, Jan 30, 2013 at 3:47 AM, Guillermo Barbero >>> > <[email protected]> wrote: >>> >> >>> >> Hi, >>> >> >>> >> I'm viewing a weird behaviour in my cassandra cluster. Most of the >>> >> warning messages are due to Heap is % full. According to this link >>> >> >>> >> (http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassndra-1-0-6-GC-query-tt7323457.html) >>> >> there are two ways to "reduce pressure": >>> >> 1. Decrease the cache sizes >>> >> 2. Increase the index interval size >>> >> >>> >> Most of the flushes are in two column families (users and messages), I >>> >> guess that's because the most mutations are there. >>> >> >>> >> I still have not applied those changes to the production environment. >>> >> Do you recommend any other meassure? Should I set specific tunning for >>> >> these two CFs? Should I check another metric? >>> >> >>> >> Additionally, the distribution of warning messages is not uniform >>> >> along the cluster. Why could cassandra be doing this? What should I do >>> >> to find out how to fix this? >>> >> >>> >> cassandra runs on a 6 node cluster of m1.xlarge machines (Amazon EC2) >>> >> the java version is the following: >>> >> java version "1.6.0_37" >>> >> Java(TM) SE Runtime Environment (build 1.6.0_37-b06) >>> >> Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) >>> >> >>> >> The cassandra system.log is resumed here (numer of messages, cassandra >>> >> node, class that reports the message, first word of the message) >>> >> 2013-01-26 >>> >> 5 cassNode0: GCInspector.java Heap >>> >> 5 cassNode0: StorageService.java Flushing >>> >> 232 cassNode2: GCInspector.java Heap >>> >> 232 cassNode2: StorageService.java Flushing >>> >> 104 cassNode3: GCInspector.java Heap >>> >> 104 cassNode3: StorageService.java Flushing >>> >> 3 cassNode4: GCInspector.java Heap >>> >> 3 cassNode4: StorageService.java Flushing >>> >> 3 cassNode5: GCInspector.java Heap >>> >> 3 cassNode5: StorageService.java Flushing >>> >> >>> >> 2013-01-27 >>> >> 2 cassNode0: GCInspector.java Heap >>> >> 2 cassNode0: StorageService.java Flushing >>> >> 3 cassNode1: GCInspector.java Heap >>> >> 3 cassNode1: StorageService.java Flushing >>> >> 189 cassNode2: GCInspector.java Heap >>> >> 189 cassNode2: StorageService.java Flushing >>> >> 104 cassNode3: GCInspector.java Heap >>> >> 104 cassNode3: StorageService.java Flushing >>> >> 1 cassNode4: GCInspector.java Heap >>> >> 1 cassNode4: StorageService.java Flushing >>> >> 1 cassNode5: GCInspector.java Heap >>> >> 1 cassNode5: StorageService.java Flushing >>> >> >>> >> 2013-01-28 >>> >> 2 cassNode0: GCInspector.java Heap >>> >> 2 cassNode0: StorageService.java Flushing >>> >> 1 cassNode1: GCInspector.java Heap >>> >> 1 cassNode1: StorageService.java Flushing >>> >> 1 cassNode2: AutoSavingCache.java Reducing >>> >> 343 cassNode2: GCInspector.java Heap >>> >> 342 cassNode2: StorageService.java Flushing >>> >> 181 cassNode3: GCInspector.java Heap >>> >> 181 cassNode3: StorageService.java Flushing >>> >> 4 cassNode4: GCInspector.java Heap >>> >> 4 cassNode4: StorageService.java Flushing >>> >> 3 cassNode5: GCInspector.java Heap >>> >> 3 cassNode5: StorageService.java Flushing >>> >> >>> >> 2013-01-29 >>> >> 2 cassNode0: GCInspector.java Heap >>> >> 2 cassNode0: StorageService.java Flushing >>> >> 3 cassNode1: GCInspector.java Heap >>> >> 3 cassNode1: StorageService.java Flushing >>> >> 156 cassNode2: GCInspector.java Heap >>> >> 156 cassNode2: StorageService.java Flushing >>> >> 71 cassNode3: GCInspector.java Heap >>> >> 71 cassNode3: StorageService.java Flushing >>> >> 2 cassNode4: GCInspector.java Heap >>> >> 2 cassNode4: StorageService.java Flushing >>> >> 2 cassNode5: GCInspector.java Heap >>> >> 1 cassNode5: Memtable.java setting >>> >> 2 cassNode5: StorageService.java Flushing >>> >> >>> >> -- >>> >> >>> >> Guillermo Barbero - Backend Team >>> >> >>> >> Spotbros Technologies >>> > >>> > >>> > >> >> >> >> >> -- >> >> Guillermo Barbero - Backend Team >> >> Spotbros Technologies >> >> >> > > > > -- > > Guillermo Barbero - Backend Team > > Spotbros Technologies
