> And I'm still getting UnavailableException and TimedOutException when there > Cassandra daemon is doing either Compaction or Garbage collection...
Have you specifically correlated this? If so, which one, or both? GC should not cause unavailable exceptions on a healthy cluster with healthy nodes. If GC:s are causing nodes to not respond for so long that they cause exceptions you may be swapping. Have you monitored to see whether you are actively swapping in/out during GC:s (or at all for that matter)? Excessive GC pause times should be logged by Cassandra i the system log, regardless of JVM options. So, check your cassandra log from messages from GCInspector about pause times. I presume you are running with default JVM paramters? WIth respect to compaction, I'm still interested, as in my original response in this thread, what your data looks like and whether compaction is CPU bound or I/O bound. It's quite possible that compaction is having an adverse effect; if it is, I would suspect it is due to disk I/O rather than CPU load (unless you are saturating your cluster so that CPU load is the dominating factor). The primary adverse effects expected by compaction, other than additional CPU load, are: * Generation of additional I/O which will directly affect normal traffic. * Effects on the operating system buffer cache may increase the I/O load resulting from normal traffic (thus making the preceding issue even more significant). Are the nodes that are slow to respond during compaction I/O bound? Check with for example "iostat -x 1" (utilization value and average queue size being the most important columns). There is some work planned/happening towards lessening the impact of compactions, and it would be of interest to know the circumstances of compaction problems that people do have. -- / Peter Schuller
