At Netflix we rotate the major compactions around the cluster, don't run them all at once. We also either take that node out of client traffic so it doesn't get used as a coordinator or use the Astyanax client that is latency and token aware to steer traffic to the other replicas.
We are running on EC2 with lots of CPU and RAM but only two internal disk spindles, so if you have lots of IOPS available in your config, you may find that it doesn't affect read times much. Adrian On Thu, Feb 2, 2012 at 11:44 PM, Peter Schuller <peter.schul...@infidyne.com> wrote: >> If every node in the cluster is running major compaction, would it be able to >> answer any read request? And is it wise to write anything to a cluster >> while it's doing major compaction? > > Compaction is something that is supposed to be continuously running in > the background. As noted, it will have a performance impact in that it > (1) generates I/O, (2) leads to cache eviction, and (if you're CPU > bound rather than disk bound) (3) adds CPU load. > > But there is no intention that clients should have to care about > compaction; it's to be viewed as a background operation continuously > happening. A good rule of thumb is that an individual node should be > able to handle your traffic when doing compaction; you don't want to > be in the position where you're just barely dealing with the traffic, > and a node doing compaction not being able to handle it. > > -- > / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
