> Is there any way in which the writes can be made pretty slow on different > nodes. Ideally I would like data to be written on one node and eventually > replicating across other nodes I dont really need a real time update, so can > pretty much live with slow writes. Replicating "inside" the mutation request is a core feature of cassandra.
You can hack something by disabling the the gossip on a node and doing the inserts on it (at CL One). Re-enable gossip and let HH send the data to the other nodes, or disable HH and use repair to distribute the changes. HH will be less resource intensive. > 1250.188: [Full GC [PSYoungGen: 76825K->0K(571648K)] [PSOldGen: > 7356362K->2356764K(7569408K)] 7433188K->2356764K(8141056K) [PSPermGen: > 32579K->32579K(61056K)], 8.1019330 secs] [Times: user=8.10 sys=0.00, > real=8.10 secs] What JVM are you using and what are the JVM options ? (The info I can find about PSYoungGen suggest it's pretty old. ) Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 5/03/2012, at 1:13 AM, Shubham Srivastava wrote: > Tried all the possible options and nothing actually seems to work. > > Was trying to get insights of where exactly problem is arising when writes on > done on one node and read on another . I found that GC gets triggered when > writes are done on the other node through RowMutationVerbHandler. > > Settings > > 1.I tested this on two node setup with RF:2 and Read CL:1. > 2.Also heap is of 8G and Xmn:800M on both the nodes with 4Cores. > 3.I am using concurrent_write:32 as default (8 * core). > 4. -Dcassandra.compaction.priority=1 > 5.No explicit GC settings commented the one in solandra-env.sh > 6.in_memory_compaction_limit_in_mb: 1 > 7.read repair:0.1 > 8.concurrent_compactors: 1 > > > Is there any way in which the writes can be made pretty slow on different > nodes. Ideally I would like data to be written on one node and eventually > replicating across other nodes I dont really need a real time update, so can > pretty much live with slow writes. > > Sharing the cassandra and GC logs as below > > ==================== > > > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf7365617263686669656c64efbfbf64657320', > modifications=[ColumnFamily(TI [357025:false:4@1330861061454,])]) applied. > Sending response to 14541197@/10.86.29.21 > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='L', > key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365', > modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 Table.java (line 387) > applying mutation of row > 686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365 > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365', > modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) applied. > Sending response to 14541198@/10.86.29.21 > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='L', > key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573', > modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 Table.java (line 387) > applying mutation of row > 686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573 > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573', > modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) applied. > Sending response to 14541199@/10.86.29.21 > DEBUG [248896865@qtp-1257398760-2] 2012-03-04 17:07:51,572 > SolrIndexReader.java (line 928) getCoreCacheKey() - start > DEBUG [MutationStage:32] 2012-03-04 17:07:51,426 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf636f756e74727953636f7265', > modifications=[ColumnFamily(FC [356996:false:4@1330861059172,])]) applied. > Sending response to 14532390@/10.86.29.21 > DEBUG [1568261127@qtp-1257398760-14] 2012-03-04 17:07:51,425 > SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start > DEBUG [MutationStage:2] 2012-03-04 17:07:51,425 Table.java (line 387) > applying mutation of row 686f74656c737e30efbfbf6e616d65efbfbf6c61 > DEBUG [1568261127@qtp-1257398760-14] 2012-03-04 17:07:52,015 > SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start > DEBUG [1861954021@qtp-1257398760-11] 2012-03-04 17:07:51,425 > StorageProxy.java (line 696) Read: 0 ms. > INFO [ScheduledTasks:1] 2012-03-04 17:07:51,425 GCInspector.java (line 123) > GC for PS Scavenge: 443 ms for 1 collections, 5983756216 used; max is > 8322088960 > > > > ============= > > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf7365617263686669656c64efbfbf64657320', > modifications=[ColumnFamily(TI [357025:false:4@1330861061454,])]) applied. > Sending response to 14541197@/10.86.29.21 > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='L', > key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365', > modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 Table.java (line 387) > applying mutation of row > 686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365 > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf49726f6e696e672053657276696365', > modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) applied. > Sending response to 14541198@/10.86.29.21 > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='L', > key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573', > modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 Table.java (line 387) > applying mutation of row > 686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573 > DEBUG [MutationStage:5] 2012-03-04 17:07:52,014 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf4c61756e647279205365727669636573', > modifications=[ColumnFamily(TI [357025:false:2@1330861061454,])]) applied. > Sending response to 14541199@/10.86.29.21 > DEBUG [248896865@qtp-1257398760-2] 2012-03-04 17:07:51,572 > SolrIndexReader.java (line 928) getCoreCacheKey() - start > DEBUG [MutationStage:32] 2012-03-04 17:07:51,426 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf636f756e74727953636f7265', > modifications=[ColumnFamily(FC [356996:false:4@1330861059172,])]) applied. > Sending response to 14532390@/10.86.29.21 > DEBUG [1568261127@qtp-1257398760-14] 2012-03-04 17:07:51,425 > SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start > DEBUG [MutationStage:2] 2012-03-04 17:07:51,425 Table.java (line 387) > applying mutation of row 686f74656c737e30efbfbf6e616d65efbfbf6c61 > DEBUG [1568261127@qtp-1257398760-14] 2012-03-04 17:07:52,015 > SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start > DEBUG [1861954021@qtp-1257398760-11] 2012-03-04 17:07:51,425 > StorageProxy.java (line 696) Read: 0 ms. > INFO [ScheduledTasks:1] 2012-03-04 17:07:51,425 GCInspector.java (line 123) > GC for PS Scavenge: 443 ms for 1 collections, 5983756216 used; max is > 8322088960 > > > > ====================== > > DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='L', > key='686f74656c737e30efbfbf636f756e74727953636f7265efbfbfc2800000', > modifications=[ColumnFamily(TI [357932:false:2@1330861128691,])]) > DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 Table.java (line 387) > applying mutation of row > 686f74656c737e30efbfbf636f756e74727953636f7265efbfbfc2800000 > DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf636f756e74727953636f7265efbfbfc2800000', > modifications=[ColumnFamily(TI [357932:false:2@1330861128691,])]) applied. > Sending response to 14863674@/10.86.29.21 > DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='L', > key='686f74656c737e30efbfbf6e756d4f664e6967687473efbfbf2001000000000000000000', > modifications=[ColumnFamily(TI [357932:false:3@1330861128691,])]) > DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 Table.java (line 387) > applying mutation of row > 686f74656c737e30efbfbf6e756d4f664e6967687473efbfbf2001000000000000000000 > DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf6e756d4f664e6967687473efbfbf2001000000000000000000', > modifications=[ColumnFamily(TI [357932:false:3@1330861128691,])]) applied. > Sending response to 14863675@/10.86.29.21 > DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='L', > key='686f74656c737e30efbfbf6164647231', modifications=[ColumnFamily(FC > [357932:false:2@1330861128691,])]) > DEBUG [MutationStage:17] 2012-03-04 17:08:58,898 Table.java (line 387) > applying mutation of row 686f74656c737e30efbfbf6164647231 > INFO [ScheduledTasks:1] 2012-03-04 17:08:57,465 StatusLogger.java (line 65) > MiscStage 0 0 0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,899 StatusLogger.java (line 65) > InternalResponseStage 0 0 0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 65) > HintedHandoff 0 0 0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 69) > CompactionManager n/a 0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 81) > MessagingService n/a 0,193 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 85) > ColumnFamily Memtable ops,data Row cache size/cap Key cache > size/cap > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 88) > system.NodeIdInfo 0,0 0/0 > 0/1 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 88) > system.IndexInfo 0,0 0/0 > 0/1 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 88) > system.LocationInfo 0,0 0/0 > 3/3 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,900 StatusLogger.java (line 88) > system.Versions 3,103 0/0 > 0/1 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) > system.Migrations 0,0 0/0 > 0/2 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) > system.HintsColumnFamily 0,0 0/0 > 0/1 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) > system.Schema 0,0 0/0 > 2/2 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) > L.FC 333959,109606980 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) > L.TI 26707,17617780 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) > L.TL 565188,54888553 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) > L.SI 5498,6575970 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) > L.Docs 260652,95271408 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) > LH.SavedHotelsInverted 0,0 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,901 StatusLogger.java (line 88) > LH.SavedHotels 0,0 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) > LH.LastViewedHotelsInverted 1,56 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) > LH.SavedSearchRequest 5,7308 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) > LH.HotelMessage 0,0 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) > LH.UserPrefrences 10,357 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) > LH.HotelTariffs 0,0 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) > LH.LastViewedHotels 1,56 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,902 StatusLogger.java (line 88) > LH.BookedHotels 0,0 0/0 > 0/0 > INFO [ScheduledTasks:1] 2012-03-04 17:08:58,903 GCInspector.java (line 123) > GC for PS Scavenge: 859 ms for 3 collections, 2983178264 used; max is > 8310358016 > DEBUG [476953411@qtp-1257398760-13] 2012-03-04 17:08:58,918 > SolrIndexReader.java (line 517) docFreq(Term) - end > > > > > ======================= > > > DEBUG [MutationStage:30] 2012-03-04 17:09:01,473 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf7365617263686669656c64efbfbf6465207061726973', > modifications=[ColumnFamily(TI [358128:false:4@1330861130798,])]) applied. > Sending response to 14871005@/10.86.29.21 > DEBUG [MutationStage:30] 2012-03-04 17:09:01,473 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='L', > key='686f74656c737e30efbfbf6461746575706461746564efbfbf2001000000265d64493114', > modifications=[ColumnFamily(TI [357958:true:4@1330861130944,])]) > DEBUG [MutationStage:30] 2012-03-04 17:09:01,473 Table.java (line 387) > applying mutation of row > 686f74656c737e30efbfbf6461746575706461746564efbfbf2001000000265d64493114 > DEBUG [1512601443@qtp-1257398760-0] 2012-03-04 17:09:01,387 > SolrIndexReader.java (line 370) maxDoc() - start > DEBUG [1512601443@qtp-1257398760-0] 2012-03-04 17:09:01,474 > SolrIndexReader.java (line 375) maxDoc() - end > DEBUG [1512601443@qtp-1257398760-0] 2012-03-04 17:09:01,474 > SolrIndexReader.java (line 370) maxDoc() - start > DEBUG [1512601443@qtp-1257398760-0] 2012-03-04 17:09:01,474 > SolrIndexReader.java (line 375) maxDoc() - end > DEBUG [1435779426@qtp-1257398760-8] 2012-03-04 17:09:01,204 > SolrIndexSearcher.java (line 557) doc(int, Set<String>) - start > DEBUG [MutationStage:11] 2012-03-04 17:09:01,201 Table.java (line 387) > applying mutation of row > 686f74656c737e30efbfbf7365617263686669656c64efbfbf76696c6c652064 > DEBUG [MutationStage:11] 2012-03-04 17:09:01,474 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf7365617263686669656c64efbfbf76696c6c652064', > modifications=[ColumnFamily(TI [357955:false:4@1330861130682,])]) applied. > Sending response to 14870425@/10.86.29.21 > DEBUG [MutationStage:28] 2012-03-04 17:09:01,190 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf6c61746c6e675f305f636f6f7264696e617465efbfbf400c021f2d5f', > modifications=[ColumnFamily(TI [357971:false:4@1330861128965,])]) applied. > Sending response to 14865348@/10.86.29.21 > DEBUG [MutationStage:1] 2012-03-04 17:09:01,190 Table.java (line 387) > applying mutation of row 686f74656c737e30efbfbf3537363433 > DEBUG [412581934@qtp-1257398760-16] 2012-03-04 17:09:01,127 > CustomQueryComponent.java (line 116) prepare(ResponseBuilder) - end > DEBUG [7421394@qtp-1257398760-22] 2012-03-04 17:09:01,047 StorageProxy.java > (line 696) Read: 1477 ms. > DEBUG [2030027307@qtp-1257398760-21] 2012-03-04 17:09:01,031 > SolrIndexSearcher.java (line 2490) setDocSet(DocSet) - start > DEBUG [2030027307@qtp-1257398760-21] 2012-03-04 17:09:01,475 > SolrIndexSearcher.java (line 2499) setDocSet(DocSet) - end > INFO [ScheduledTasks:1] 2012-03-04 17:09:00,937 GCInspector.java (line 123) > GC for PS Scavenge: 351 ms for 1 collections, 3051670552 used; max is > 8329953280 > > > > ========== > > DEBUG [MutationStage:24] 2012-03-04 17:09:03,382 Table.java (line 387) > applying mutation of row 686f74656c737e30efbfbf6164647231efbfbf6465 > DEBUG [MutationStage:24] 2012-03-04 17:09:03,382 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf6164647231efbfbf6465', > modifications=[ColumnFamily(TI [358305:false:4@1330861132989,])]) applied. > Sending response to 14880403@/10.86.29.21 > DEBUG [MutationStage:18] 2012-03-04 17:09:03,141 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf6e616d65efbfbf76616c656e74696e', > modifications=[ColumnFamily(TI [358305:false:5@1330861132989,])]) applied. > Sending response to 14880326@/10.86.29.21 > DEBUG [MutationStage:1] 2012-03-04 17:09:03,052 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf4661782f50686f746f636f7079696e67', > modifications=[ColumnFamily(TI [358305:false:2@1330861132990,])]) applied. > Sending response to 14880195@/10.86.29.21 > DEBUG [MutationStage:1] 2012-03-04 17:09:03,382 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='L', > key='686f74656c737e30efbfbf6c61746c6e675f315f636f6f7264696e617465efbfbf38177e4a374800', > modifications=[ColumnFamily(TI [358252:true:4@1330861132886,])]) > DEBUG [MutationStage:1] 2012-03-04 17:09:03,382 Table.java (line 387) > applying mutation of row > 686f74656c737e30efbfbf6c61746c6e675f315f636f6f7264696e617465efbfbf38177e4a374800 > DEBUG [MutationStage:11] 2012-03-04 17:09:03,037 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf686f74656c666163696c69747964657461696cefbfbf5065747320416c6c6f776564', > modifications=[ColumnFamily(TI [358252:true:4@1330861132886,])]) applied. > Sending response to 14880194@/10.86.29.21 > DEBUG [728126393@qtp-1257398760-17] 2012-03-04 17:09:03,037 StorageProxy.java > (line 696) Read: 1396 ms. > DEBUG [MutationStage:28] 2012-03-04 17:09:03,024 RowMutationVerbHandler.java > (line 56) RowMutation(keyspace='L', > key='686f74656c737e30efbfbf686f74656c666163696c697479efbfbf44697361626c656420466163696c6974696573', > modifications=[ColumnFamily(TI [358252:true:4@1330861132886,])]) applied. > Sending response to 14880157@/10.86.29.21 > DEBUG [MutationStage:28] 2012-03-04 17:09:03,384 RowMutationVerbHandler.java > (line 44) Applying RowMutation(keyspace='L', > key='686f74656c737e30efbfbf67726f7373546172696666efbfbf2001000000000000000000', > modifications=[ColumnFamily(TI [358305:false:3@1330861132989,])]) > DEBUG [MutationStage:28] 2012-03-04 17:09:03,384 Table.java (line 387) > applying mutation of row > 686f74656c737e30efbfbf67726f7373546172696666efbfbf2001000000000000000000 > DEBUG [MutationStage:23] 2012-03-04 17:09:03,012 Table.java (line 387) > applying mutation of row 686f74656c737e30efbfbf6665617475726573746174efbfbf4e > INFO [ScheduledTasks:1] 2012-03-04 17:09:02,938 GCInspector.java (line 123) > GC for PS Scavenge: 319 ms for 1 collections, 3148126656 used; max is > 8329166848 > > > ===== GC Logs===== > > 1248.456: [GC > Desired survivor size 266403840 bytes, new threshold 1 (max 15) > [PSYoungGen: 461285K->63226K(548736K)] 7589727K->7360609K(8118144K), > 0.2978660 secs] [Times: user=1.15 sys=0.01, real=0.29 secs] > Total time for which application threads were stopped: 0.2985800 seconds > 1250.024: [GC > Desired survivor size 253493248 bytes, new threshold 1 (max 15) > [PSYoungGen: 341498K->76825K(571648K)] 7638881K->7433188K(8141056K), > 0.1632840 secs] [Times: user=0.64 sys=0.00, real=0.16 secs] > 1250.188: [Full GC [PSYoungGen: 76825K->0K(571648K)] [PSOldGen: > 7356362K->2356764K(7569408K)] 7433188K->2356764K(8141056K) [PSPermGen: > 32579K->32579K(61056K)], 8.1019330 secs] [Times: user=8.10 sys=0.00, > real=8.10 secs] > Total time for which application threads were stopped: 8.2661710 seconds > Total time for which application threads were stopped: 0.0056460 seconds > 1260.267: [GC > Desired survivor size 246808576 bytes, new threshold 1 (max 15) > [PSYoungGen: 311488K->119239K(559040K)] 2668252K->2476003K(8128448K), > 0.1174730 secs] [Times: user=0.42 sys=0.00, real=0.12 secs] > Total time for which application threads were stopped: 0.1181790 seconds > Total time for which application threads were stopped: 0.0071000 seconds > 1262.020: [GC > Desired survivor size 247529472 bytes, new threshold 1 (max 15) > [PSYoungGen: 430727K->138637K(576768K)] 2787491K->2612633K(8146176K), > 0.2413110 secs] [Times: user=0.84 sys=0.00, real=0.25 secs] > Total time for which application threads were stopped: 0.2472850 seconds > Total time for which application threads were stopped: 0.0012250 seconds > 1263.959: [GC > Desired survivor size 236191744 bytes, new threshold 1 (max 15) > [PSYoungGen: 474381K->85226K(577472K)] 2948377K->2686551K(8146880K), > 0.2765150 secs] [Times: user=1.06 sys=0.00, real=0.28 secs] > Total time for which application threads were stopped: 0.2811880 seconds > 1265.567: [GC > Desired survivor size 257294336 bytes, new threshold 1 (max 15) > [PSYoungGen: 420970K->191373K(547328K)] 3022295K->2873771K(8116736K), > 0.2006350 secs] [Times: user=0.79 sys=0.00, real=0.20 secs] > Total time for which application threads were stopped: 0.2067200 seconds > Total time for which application threads were stopped: 0.0057330 seconds > 1267.417: [GC > Desired survivor size 264765440 bytes, new threshold 1 (max 15) > [PSYoungGen: 508045K->167618K(484352K)] 3190443K->3040491K(8053760K), > 0.3643920 secs] [Times: user=1.25 sys=0.00, real=0.36 secs] > Total time for which application threads were stopped: 0.3704240 seconds > 1268.904: [GC > Desired survivor size 252641280 bytes, new threshold 1 (max 15) > [PSYoungGen: 484290K->100796K(572480K)] 3357163K->3140418K(8141888K), > 0.3179310 secs] [Times: user=1.25 sys=0.00, real=0.31 secs] > Total time for which application threads were stopped: 0.3241000 seconds > Total time for which application threads were stopped: 0.0057010 seconds > 1270.689: [GC > Desired survivor size 242483200 bytes, new threshold 1 (max 15) > [PSYoungGen: 414716K->123054K(560640K)] 3454338K->3258021K(8130048K), > 0.1692080 secs] > > Regards, > Shubham > > > From: aaron morton [aa...@thelastpickle.com] > Sent: Thursday, February 23, 2012 2:00 AM > To: user@cassandra.apache.org > Subject: Re: Maximum Row Size in Cassandra : Potential Bottleneck > > I'm not familiar with the solandra data model, but they are wide rows. > > In the past I've done the following to reduce GC pressure with wide rows > where I thought GC was associated with compaction. It's heavy handed and may > put pressure on the IO system. Once things are stabilised you may want to > undo some of the changes. > > * modify cassandra-env.sh to make sure the new heap is no more than 800M . > * modify cassandra.yaml and change concurrent_compactors to 2 > * modify cassandra.yaml and reduce in_memory_compaction_limit_in_mb to 32 > > Then start watching the logs. > > Hope that helps. > > > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 21/02/2012, at 10:39 PM, Shubham Srivastava wrote: > >> Hi, >> >> I am using cassandra0.8.6 for saving inverted indexes through Solr(Solr3.3 + >> Cassandra). >> >> I am having problem while doing writes which causes lot of GC activity >> where CPU usage increases + heap size usage and my nodes goes on a >> standstill state dropping most of reads and writes as per cfstats given >> below. >> >> Looking at the row sizes of some of CF’s it seems to be on the higher side >> right from average row size of TL from 77 mb with max row size as 186 >> mb.Typically TL family is used for writing various terms of an indexed field >> and so on. >> >> While doing writes to Cassandra we update/write back this column family + >> others as well. Is the row size like of TL normal or is it on the upper side. >> >> My system can sustain small writes which actually happened after reducing >> the in in_memory_compaction_limit_in_mb: to 2 from 64 and read_repair_chance >> to 0.1 from 1. >> >> I have a 7 node setup with RF:7 and Read CL:1 with 8 core processor and 14GB >> RAM. Hepa size is 8Gb on all the nodes. >> >> >> >> >> CFSTATS: >> >> Keyspace: system >> Read Count: 19 >> Read Latency: 8.130157894736843 ms. >> Write Count: 31 >> Write Latency: 0.03029032258064516 ms. >> Pending Tasks: 0 >> Column Family: NodeIdInfo >> SSTable count: 1 >> Space used (live): 4516 >> Space used (total): 4516 >> Number of Keys (estimate): 128 >> Memtable Columns Count: 0 >> Memtable Data Size: 0 >> Memtable Switch Count: 0 >> Read Count: 0 >> Read Latency: NaN ms. >> Write Count: 0 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache capacity: 1 >> Key cache size: 0 >> Key cache hit rate: NaN >> Row cache: disabled >> Compacted row minimum size: 87 >> Compacted row maximum size: 103 >> Compacted row mean size: 103 >> >> Column Family: HintsColumnFamily >> SSTable count: 2 >> Space used (live): 9880 >> Space used (total): 34550 >> Number of Keys (estimate): 256 >> Memtable Columns Count: 0 >> Memtable Data Size: 0 >> Memtable Switch Count: 5 >> Read Count: 10 >> Read Latency: NaN ms. >> Write Count: 28 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache capacity: 2 >> Key cache size: 2 >> Key cache hit rate: NaN >> Row cache: disabled >> Compacted row minimum size: 125 >> Compacted row maximum size: 149 >> Compacted row mean size: 149 >> >> Column Family: Schema >> SSTable count: 1 >> Space used (live): 192414 >> Space used (total): 192414 >> Number of Keys (estimate): 128 >> Memtable Columns Count: 0 >> Memtable Data Size: 0 >> Memtable Switch Count: 0 >> Read Count: 3 >> Read Latency: NaN ms. >> Write Count: 0 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache capacity: 1 >> Key cache size: 1 >> Key cache hit rate: NaN >> Row cache: disabled >> Compacted row minimum size: 104 >> Compacted row maximum size: 8239 >> Compacted row mean size: 7794 >> >> Column Family: Migrations >> SSTable count: 1 >> Space used (live): 300650 >> Space used (total): 300650 >> Number of Keys (estimate): 128 >> Memtable Columns Count: 0 >> Memtable Data Size: 0 >> Memtable Switch Count: 0 >> Read Count: 0 >> Read Latency: NaN ms. >> Write Count: 0 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache capacity: 1 >> Key cache size: 0 >> Key cache hit rate: NaN >> Row cache: disabled >> Compacted row minimum size: 263211 >> Compacted row maximum size: 315852 >> Compacted row mean size: 315852 >> >> Column Family: IndexInfo >> SSTable count: 0 >> Space used (live): 0 >> Space used (total): 0 >> Number of Keys (estimate): 0 >> Memtable Columns Count: 0 >> Memtable Data Size: 0 >> Memtable Switch Count: 0 >> Read Count: 0 >> Read Latency: NaN ms. >> Write Count: 0 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache capacity: 1 >> Key cache size: 0 >> Key cache hit rate: NaN >> Row cache: disabled >> Compacted row minimum size: 0 >> Compacted row maximum size: 0 >> Compacted row mean size: 0 >> >> Column Family: LocationInfo >> SSTable count: 2 >> Space used (live): 11496 >> Space used (total): 11496 >> Number of Keys (estimate): 256 >> Memtable Columns Count: 0 >> Memtable Data Size: 0 >> Memtable Switch Count: 2 >> Read Count: 6 >> Read Latency: NaN ms. >> Write Count: 3 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache capacity: 2 >> Key cache size: 2 >> Key cache hit rate: NaN >> Row cache: disabled >> Compacted row minimum size: 73 >> Compacted row maximum size: 310 >> Compacted row mean size: 146 >> >> ---------------- >> Keyspace: L >> Read Count: 343746 >> Read Latency: 5.11407672525644 ms. >> Write Count: 541480 >> Write Latency: 0.023806109182241266 ms. >> Pending Tasks: 0 >> Column Family: FC >> SSTable count: 4 >> Space used (live): 353742759 >> Space used (total): 370460012 >> Number of Keys (estimate): 512 >> Memtable Columns Count: 64604 >> Memtable Data Size: 65810913 >> Memtable Switch Count: 4 >> Read Count: 93 >> Read Latency: NaN ms. >> Write Count: 138640 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 104 >> Compacted row maximum size: 10090808 >> Compacted row mean size: 2650773 >> >> Column Family: Docs >> SSTable count: 9 >> Space used (live): 1645341386 >> Space used (total): 1726113393 >> Number of Keys (estimate): 356608 >> Memtable Columns Count: 65268 >> Memtable Data Size: 27459963 >> Memtable Switch Count: 4 >> Read Count: 74566 >> Read Latency: 1.995 ms. >> Write Count: 4688 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 87 >> Compacted row maximum size: 51012 >> Compacted row mean size: 5557 >> >> Column Family: SI >> SSTable count: 4 >> Space used (live): 88904425 >> Space used (total): 105431969 >> Number of Keys (estimate): 288512 >> Memtable Columns Count: 1141 >> Memtable Data Size: 767070 >> Memtable Switch Count: 4 >> Read Count: 8473 >> Read Latency: 0.615 ms. >> Write Count: 2447 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 104 >> Compacted row maximum size: 25109160 >> Compacted row mean size: 4694 >> >> Column Family: TL >> SSTable count: 5 >> Space used (live): 334547547 >> Space used (total): 348394559 >> Number of Keys (estimate): 640 >> Memtable Columns Count: 90495 >> Memtable Data Size: 8602591 >> Memtable Switch Count: 3 >> Read Count: 5236 >> Read Latency: 30.199 ms. >> Write Count: 2335 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 454827 >> Compacted row maximum size: 186563160 >> Compacted row mean size: 77287206 >> >> Column Family: TI >> SSTable count: 10 >> Space used (live): 3056043044 >> Space used (total): 3084009723 >> Number of Keys (estimate): 8494720 >> Memtable Columns Count: 181003 >> Memtable Data Size: 212200530 >> Memtable Switch Count: 4 >> Read Count: 255378 >> Read Latency: 0.962 ms. >> Write Count: 393370 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 104 >> Compacted row maximum size: 7007506 >> Compacted row mean size: 633 >> >> ---------------- >> Keyspace: LH >> Read Count: 40729 >> Read Latency: 1.1628788332637678 ms. >> Write Count: 265328 >> Write Latency: 0.01778465898812036 ms. >> Pending Tasks: 0 >> Column Family: UserPrefrences >> SSTable count: 6 >> Space used (live): 639728096 >> Space used (total): 786966376 >> Number of Keys (estimate): 2790912 >> Memtable Columns Count: 14546 >> Memtable Data Size: 7532750 >> Memtable Switch Count: 9 >> Read Count: 18874 >> Read Latency: 0.483 ms. >> Write Count: 107364 >> Write Latency: 0.015 ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 87 >> Compacted row maximum size: 372 >> Compacted row mean size: 239 >> >> Column Family: LastViewedHotels >> SSTable count: 4 >> Space used (live): 218367054 >> Space used (total): 227203324 >> Number of Keys (estimate): 1096192 >> Memtable Columns Count: 3724 >> Memtable Data Size: 5134228 >> Memtable Switch Count: 9 >> Read Count: 6085 >> Read Latency: 0.939 ms. >> Write Count: 18849 >> Write Latency: 0.022 ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 73 >> Compacted row maximum size: 51012 >> Compacted row mean size: 186 >> >> Column Family: BookedHotels >> SSTable count: 2 >> Space used (live): 8288377 >> Space used (total): 24867472 >> Number of Keys (estimate): 44928 >> Memtable Columns Count: 24 >> Memtable Data Size: 46264 >> Memtable Switch Count: 8 >> Read Count: 111 >> Read Latency: 0.025 ms. >> Write Count: 209 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 87 >> Compacted row maximum size: 35425 >> Compacted row mean size: 152 >> >> Column Family: HotelMessage >> SSTable count: 3 >> Space used (live): 683567 >> Space used (total): 2143936 >> Number of Keys (estimate): 1536 >> Memtable Columns Count: 14536 >> Memtable Data Size: 3764728 >> Memtable Switch Count: 8 >> Read Count: 4804 >> Read Latency: 2.714 ms. >> Write Count: 64967 >> Write Latency: 0.012 ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 87 >> Compacted row maximum size: 20501 >> Compacted row mean size: 496 >> >> Column Family: SavedHotels >> SSTable count: 3 >> Space used (live): 1035147 >> Space used (total): 3097633 >> Number of Keys (estimate): 5760 >> Memtable Columns Count: 8 >> Memtable Data Size: 15376 >> Memtable Switch Count: 7 >> Read Count: 3971 >> Read Latency: 0.023 ms. >> Write Count: 73 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 104 >> Compacted row maximum size: 2299 >> Compacted row mean size: 170 >> >> Column Family: SavedHotelsInverted >> SSTable count: 1 >> Space used (live): 1022963 >> Space used (total): 4108255 >> Number of Keys (estimate): 5504 >> Memtable Columns Count: 8 >> Memtable Data Size: 450 >> Memtable Switch Count: 7 >> Read Count: 3 >> Read Latency: NaN ms. >> Write Count: 73 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 104 >> Compacted row maximum size: 2299 >> Compacted row mean size: 162 >> >> Column Family: LastViewedHotelsInverted >> SSTable count: 4 >> Space used (live): 243688384 >> Space used (total): 362107541 >> Number of Keys (estimate): 1257216 >> Memtable Columns Count: 2917 >> Memtable Data Size: 4254486 >> Memtable Switch Count: 9 >> Read Count: 2454 >> Read Latency: 0.091 ms. >> Write Count: 18857 >> Write Latency: 0.018 ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 73 >> Compacted row maximum size: 51012 >> Compacted row mean size: 170 >> >> Column Family: SavedSearchRequest >> SSTable count: 7 >> Space used (live): 7125532732 >> Space used (total): 7266974627 >> Number of Keys (estimate): 1526144 >> Memtable Columns Count: 8854 >> Memtable Data Size: 23875680 >> Memtable Switch Count: 9 >> Read Count: 4427 >> Read Latency: 5.022 ms. >> Write Count: 54938 >> Write Latency: 0.026 ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 925 >> Compacted row maximum size: 4055269 >> Compacted row mean size: 4912 >> >> Column Family: HotelTariffs >> SSTable count: 1 >> Space used (live): 46306159 >> Space used (total): 46306159 >> Number of Keys (estimate): 7680 >> Memtable Columns Count: 0 >> Memtable Data Size: 0 >> Memtable Switch Count: 0 >> Read Count: 0 >> Read Latency: NaN ms. >> Write Count: 0 >> Write Latency: NaN ms. >> Pending Tasks: 0 >> Key cache: disabled >> Row cache: disabled >> Compacted row minimum size: 180 >> Compacted row maximum size: 9887 >> Compacted row mean size: 6535 >> >> >> >> >> Regards, >> Shubham Srivastava | Technical Lead - Technology Development >> +91 124 439 5428 | MakeMyTrip.com, 103 Udyog Vihar Phase 1, Gurgaon, >> Haryana - 122 016, India >> <image002.png>What's new? My Trip Rewards - An exclusive loyalty program for >> MakeMyTrip customers. >> <image003.gif> >> <image004.gif> >> Office Map >> <image005.gif> >> Facebook >> <image006.gif> >> Twitter
