A little update:

OK, after ~8 hours of GC madness and compacting node B (the one on which keyspace has disappeared) works fine. No issues noticed so far.

Node A was started with larger heap and after I turned debugging on I can see it does this:

DEBUG [MutationStage:110] 2013-04-23 06:23:44,893 AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) DEBUG [MutationStage:117] 2013-04-23 06:23:44,893 AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) DEBUG [MutationStage:73] 2013-04-23 06:23:44,893 AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) DEBUG [MutationStage:108] 2013-04-23 06:23:44,893 AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for cleaned-up value DecoratedKey(1, 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) DEBUG [MutationStage:87] 2013-04-23 06:23:44,893 AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for cleaned-up value DecoratedKey(1, 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) DEBUG [MutationStage:70] 2013-04-23 06:23:44,893 AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) DEBUG [MutationStage:86] 2013-04-23 06:23:44,892 AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for cleaned-up value DecoratedKey(1, 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) DEBUG [MutationStage:86] 2013-04-23 06:23:44,898 AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) DEBUG [MutationStage:86] 2013-04-23 06:23:44,898 AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for cleaned-up value DecoratedKey(1, 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) DEBUG [MutationStage:86] 2013-04-23 06:23:44,898 AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) DEBUG [MutationStage:74] 2013-04-23 06:23:44,892 AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for cleaned-up value DecoratedKey(1, 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) DEBUG [MutationStage:123] 2013-04-23 06:23:44,891 RowMutationVerbHandler.java (line 40) Applying mutation DEBUG [MutationStage:69] 2013-04-23 06:23:44,891 AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for cleaned-up value DecoratedKey(1, 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) DEBUG [MutationStage:67] 2013-04-23 06:23:44,891 AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) DEBUG [MutationStage:84] 2013-04-23 06:23:44,891 AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row 1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,]) DEBUG [MutationStage:83] 2013-04-23 06:23:44,890 AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for cleaned-up value DecoratedKey(1, 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,]) DEBUG [MutationStage:77] 2013-04-23 06:23:44,890 AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for cleaned-up value DecoratedKey(1, 01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx [416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,])
(...)
DEBUG [MutationStage:115] 2013-04-23 06:25:24,084 AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row android_re in ColumnFamily(DRDevices.DRDevices_item_template_idx [416e64726f69645f536174656c6c6974655f4c3430:false:0@1366633668172657,]) DEBUG [MutationStage:71] 2013-04-23 06:25:24,084 AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row android_re in ColumnFamily(DRDevices.DRDevices_item_template_idx [416e64726f69645f536174656c6c6974655f4c3430:false:0@1366633668172657,]) DEBUG [MutationStage:103] 2013-04-23 06:25:24,084 AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index entry for cleaned-up value DecoratedKey(android_re, 616e64726f69645f7265):ColumnFamily(DRDevices.DRDevices_item_template_idx [416e64726f69645f536174656c6c6974655f4c3430:true:4@1366633668172657,]) DEBUG [MutationStage:90] 2013-04-23 06:25:24,084 AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row android_re in ColumnFamily(DRDevices.DRDevices_item_template_idx [416e64726f69645f536174656c6c6974655f4c3430:false:0@1366633668172657,])

And it goes this way for ~10 minutes so far and it doesn't seem like it's going to end. I think one important thing to mention here is that the DRDevices CF is ~2MB large, so I guess in this time C* should be able to be fully rewrite it at least a few times... ;-)



M.

Reply via email to