I was surprised the other day to discover that this was a cluster-wide setting. Why does that make sense?
In a heterogeneous cassandra deployment, say I have some old servers running spinning disks and I'm bringing on more nodes that perhaps utilize SSD. I want to have different compaction throttling on different nodes to minimize read impact times. I can already balance data ownership through either token allocation or vnode counts. Also, as I increase my node count, I technically also have to increase my compaction_throughput which would require a rolling restart across the cluster.
