That sounds horrible. The log messages seem fine to me. It's handling eventually updating the secondary indexes.
Good luck. ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 23/04/2013, at 6:34 PM, Michal Michalski <mich...@opera.com> wrote: > A little update: > > OK, after ~8 hours of GC madness and compacting node B (the one on which > keyspace has disappeared) works fine. No issues noticed so far. > > Node A was started with larger heap and after I turned debugging on I can see > it does this: > > DEBUG [MutationStage:110] 2013-04-23 06:23:44,893 > AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in > ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) > DEBUG [MutationStage:117] 2013-04-23 06:23:44,893 > AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in > ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) > DEBUG [MutationStage:73] 2013-04-23 06:23:44,893 > AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in > ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) > DEBUG [MutationStage:108] 2013-04-23 06:23:44,893 > AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for > cleaned-up value DecoratedKey(1, > 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) > DEBUG [MutationStage:87] 2013-04-23 06:23:44,893 > AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for > cleaned-up value DecoratedKey(1, > 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) > DEBUG [MutationStage:70] 2013-04-23 06:23:44,893 > AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in > ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) > DEBUG [MutationStage:86] 2013-04-23 06:23:44,892 > AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for > cleaned-up value DecoratedKey(1, > 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) > DEBUG [MutationStage:86] 2013-04-23 06:23:44,898 > AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in > ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) > DEBUG [MutationStage:86] 2013-04-23 06:23:44,898 > AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for > cleaned-up value DecoratedKey(1, > 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) > DEBUG [MutationStage:86] 2013-04-23 06:23:44,898 > AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in > ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) > DEBUG [MutationStage:74] 2013-04-23 06:23:44,892 > AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for > cleaned-up value DecoratedKey(1, > 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) > DEBUG [MutationStage:123] 2013-04-23 06:23:44,891 RowMutationVerbHandler.java > (line 40) Applying mutation > DEBUG [MutationStage:69] 2013-04-23 06:23:44,891 > AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for > cleaned-up value DecoratedKey(1, > 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) > DEBUG [MutationStage:67] 2013-04-23 06:23:44,891 > AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in > ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) > DEBUG [MutationStage:84] 2013-04-23 06:23:44,891 > AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in > ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) > DEBUG [MutationStage:83] 2013-04-23 06:23:44,890 > AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for > cleaned-up value DecoratedKey(1, > 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) > DEBUG [MutationStage:77] 2013-04-23 06:23:44,890 > AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for > cleaned-up value DecoratedKey(1, > 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx > [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) > (...) > DEBUG [MutationStage:115] 2013-04-23 06:25:24,084 > AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row > android_re in ColumnFamily(DRDevices.DRDevices_item_template_idx > [416e64726f69645f536174656c6c6974655f4c3430:false:0@1366633668172657,]) > DEBUG [MutationStage:71] 2013-04-23 06:25:24,084 > AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row > android_re in ColumnFamily(DRDevices.DRDevices_item_template_idx > [416e64726f69645f536174656c6c6974655f4c3430:false:0@1366633668172657,]) > DEBUG [MutationStage:103] 2013-04-23 06:25:24,084 > AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for > cleaned-up value DecoratedKey(android_re, > 616e64726f69645f7265):ColumnFamily(DRDevices.DRDevices_item_template_idx > [416e64726f69645f536174656c6c6974655f4c3430:true:4@1366633668172657,]) > DEBUG [MutationStage:90] 2013-04-23 06:25:24,084 > AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row > android_re in ColumnFamily(DRDevices.DRDevices_item_template_idx > [416e64726f69645f536174656c6c6974655f4c3430:false:0@1366633668172657,]) > > And it goes this way for ~10 minutes so far and it doesn't seem like it's > going to end. I think one important thing to mention here is that the > DRDevices CF is ~2MB large, so I guess in this time C* should be able to be > fully rewrite it at least a few times... ;-) > > > > M.
