Is there some deep architectural reason why compaction can't be replication-aware?
What I mean is, if one node is doing compaction, its replicas shouldn't be doing compaction at the same time. Or, at least a quorum of nodes should be available at all times. For example, if RF=3, and one node is doing compaction, the nodes to its right and left in the ring should wait on compaction until that node is done. Of course, my real problem is that compaction makes a node pretty much unavailable. If we can fix that problem then this is not necessary.
