Hi, We are running a 6 Node cluster in 2 DC on DSC 3.0.3, with 3 Node each. One of the node was showing UNREACHABLE on other nodes in nodetool describecluster and on that node it was showing all others UNREACHABLE and as a measure we restarted the node.
But on doing that it is stuck possibly at with these messages in system.log: DEBUG [SlabPoolCleaner] 2016-05-29 14:07:28,156 ColumnFamilyStore.java:829 - Enqueuing flush of batches: 226784704 (11%) on-heap, 0 (0%) off-heap DEBUG [main] 2016-05-29 14:07:28,576 CommitLogReplayer.java:415 - Replaying /commitlog/data/CommitLog-6-1464508993391.log (CL version 6, messaging version 10, compression null) DEBUG [main] 2016-05-29 14:07:28,781 ColumnFamilyStore.java:829 - Enqueuing flush of batches: 207333510 (10%) on-heap, 0 (0%) off-heap MemtablePostFlush / MemtableFlushWriter stages where it is stuck with pending messages. This has been the status of them as per *nodetool tpstats *for long. MemtablePostFlush Active - 1 pending - 52 completed - 16 MemtableFlushWriter Active - 2 pending - 13 completed - 15 We restarted the node by setting log level to TRACE but in vain. What could be a possible contingency plan in such a scenario? Best Regards, Bhuvan
