Hard to say much without knowing about the cassandra configurations. Yes compactions/GC's could skipe the CPU, I had similar behavior with my setup.
-VK On Fri, Dec 6, 2013 at 7:40 PM, srmore <comom...@gmail.com> wrote: > We have a 3 node cluster running cassandra 1.2.12, they are pretty big > machines 64G ram with 16 cores, cassandra heap is 8G. > > The interesting observation is that, when I send traffic to one node its > performance is 2x more than when I send traffic to all the nodes. We ran > 1.0.11 on the same box and we observed a slight dip but not half as seen > with 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing > CL to ONE make a slight improvement but not much. > > The read_Repair_chance is 0.1. We see some compactions running. > > following is my iostat -x output, sda is the ssd (for commit log) and sdb > is the spinner. > > avg-cpu: %user %nice %system %iowait %steal %idle > 66.46 0.00 8.95 0.01 0.00 24.58 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz > avgqu-sz await svctm %util > sda 0.00 27.60 0.00 4.40 0.00 256.00 58.18 > 0.01 2.55 1.32 0.58 > sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 > sda2 0.00 27.60 0.00 4.40 0.00 256.00 58.18 > 0.01 2.55 1.32 0.58 > sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 > sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 > dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 > dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 > 0.00 5.33 2.67 0.16 > dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 > dm-3 0.00 0.00 0.00 24.80 0.00 198.40 8.00 > 0.24 9.80 0.13 0.32 > dm-4 0.00 0.00 0.00 6.60 0.00 52.80 8.00 > 0.01 1.36 0.55 0.36 > dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 > dm-6 0.00 0.00 0.00 24.80 0.00 198.40 8.00 > 0.29 11.60 0.13 0.32 > > > > I can see I am cpu bound here but couldn't figure out exactly what is > causing it, is this caused by GC or Compaction ? I am thinking it is > compaction, I see a lot of context switches and interrupts in my vmstat > output. > > I don't see GC activity in the logs but see some compaction activity. Has > anyone seen this ? or know what can be done to free up the CPU. > > Thanks, > Sandeep > > >
