A little update:
OK, after ~8 hours of GC madness and compacting node B (the one on which
keyspace has disappeared) works fine. No issues noticed so far.
Node A was started with larger heap and after I turned debugging on I
can see it does this:
DEBUG [MutationStage:110] 2013-04-23 06:23:44,893
AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row
1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,])
DEBUG [MutationStage:117] 2013-04-23 06:23:44,893
AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row
1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,])
DEBUG [MutationStage:73] 2013-04-23 06:23:44,893
AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row
1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,])
DEBUG [MutationStage:108] 2013-04-23 06:23:44,893
AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index
entry for cleaned-up value DecoratedKey(1,
01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,])
DEBUG [MutationStage:87] 2013-04-23 06:23:44,893
AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index
entry for cleaned-up value DecoratedKey(1,
01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,])
DEBUG [MutationStage:70] 2013-04-23 06:23:44,893
AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row
1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,])
DEBUG [MutationStage:86] 2013-04-23 06:23:44,892
AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index
entry for cleaned-up value DecoratedKey(1,
01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,])
DEBUG [MutationStage:86] 2013-04-23 06:23:44,898
AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row
1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,])
DEBUG [MutationStage:86] 2013-04-23 06:23:44,898
AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index
entry for cleaned-up value DecoratedKey(1,
01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,])
DEBUG [MutationStage:86] 2013-04-23 06:23:44,898
AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row
1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,])
DEBUG [MutationStage:74] 2013-04-23 06:23:44,892
AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index
entry for cleaned-up value DecoratedKey(1,
01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,])
DEBUG [MutationStage:123] 2013-04-23 06:23:44,891
RowMutationVerbHandler.java (line 40) Applying mutation
DEBUG [MutationStage:69] 2013-04-23 06:23:44,891
AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index
entry for cleaned-up value DecoratedKey(1,
01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,])
DEBUG [MutationStage:67] 2013-04-23 06:23:44,891
AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row
1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,])
DEBUG [MutationStage:84] 2013-04-23 06:23:44,891
AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row
1 in ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:false:0@1366633655364522,])
DEBUG [MutationStage:83] 2013-04-23 06:23:44,890
AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index
entry for cleaned-up value DecoratedKey(1,
01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,])
DEBUG [MutationStage:77] 2013-04-23 06:23:44,890
AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index
entry for cleaned-up value DecoratedKey(1,
01):ColumnFamily(DRDevices.DRDevices_touchscreen_idx
[416e64726f69645f48494c4956455f48493453:true:4@1366633655364522,])
(...)
DEBUG [MutationStage:115] 2013-04-23 06:25:24,084
AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row
android_re in ColumnFamily(DRDevices.DRDevices_item_template_idx
[416e64726f69645f536174656c6c6974655f4c3430:false:0@1366633668172657,])
DEBUG [MutationStage:71] 2013-04-23 06:25:24,084
AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row
android_re in ColumnFamily(DRDevices.DRDevices_item_template_idx
[416e64726f69645f536174656c6c6974655f4c3430:false:0@1366633668172657,])
DEBUG [MutationStage:103] 2013-04-23 06:25:24,084
AbstractSimplePerColumnSecondaryIndex.java (line 100) removed index
entry for cleaned-up value DecoratedKey(android_re,
616e64726f69645f7265):ColumnFamily(DRDevices.DRDevices_item_template_idx
[416e64726f69645f536174656c6c6974655f4c3430:true:4@1366633668172657,])
DEBUG [MutationStage:90] 2013-04-23 06:25:24,084
AbstractSimplePerColumnSecondaryIndex.java (line 118) applying index row
android_re in ColumnFamily(DRDevices.DRDevices_item_template_idx
[416e64726f69645f536174656c6c6974655f4c3430:false:0@1366633668172657,])
And it goes this way for ~10 minutes so far and it doesn't seem like
it's going to end. I think one important thing to mention here is that
the DRDevices CF is ~2MB large, so I guess in this time C* should be
able to be fully rewrite it at least a few times... ;-)
M.
