> I am just running simple test in 6 node cassandra 4 GB heap, 96 GB RAM and > 12 core per host. I am inserting 1M rows with avg col size of 250k. I keep > getting "Dropped mutation" messages in logs. Not sure how to troubleshoot or > tune it.
Average col size of 250k - that sounds to me like you're almost certainly going to be bottlenecking on disk I/O. Saturating your "active" in the mutation stage and building up pending is consistent with simply writing faster than writes can be handled. At first I was skeptical and figured maybe something was wrong, but upon re-reading and spotting your 250k column size - it's really easy to have a stress client saturate nodes with data sizes that large. The first thing I would do is to just look at what's going on on the system. For example, just run "iostat -x -k 1" on the machines and see whether you're completely disk bound or not. I suspect you are, and that the effects you're seeing is simply the result of that. However that would depend on how many mutations per second you're actually sending. But if you're using out-of-the-box stress.py without rate limiting and using a column size of 250k, I am not at all surprised that you're easily able to saturate your nodes. -- / Peter Schuller
