Re: CPU high load

2016-08-05 Thread Aoi Kadoya
Thank you, Alain. There was no frequent GC nor compaction so it have been a mystery,however, once I stopped chef-client(we're managing the cluster though chef-cookbook), the load was eased for almost all of the servers. so we're now refactoring our cookbook, in the meanwhile, we also decided to re

Re: CPU high load

2016-07-20 Thread Alain RODRIGUEZ
Hi Aoi, > since few weeks > ago, all of the cluster nodes are hitting avg. 15-20 cpu load. > These nodes are running on VMs(VMware vSphere) that have 8vcpu > (1core/socket)-16 vRAM.(JVM options : -Xms8G -Xmx8G -Xmn800M) I take my chance, a few ideas / questions below: - What Cassandra versi

Re: CPU high load

2016-07-13 Thread Aoi Kadoya
Hi Romain, No, I don't think we upgraded cassandra version or changed any of those schema elements. After I realized this high load issue, I found that some of the tables have a shorter gc_grace_seconds(1day) than the rest and because it seemed causing constant compaction cycles, I have changed th

Re: CPU high load

2016-07-13 Thread Romain Hardouin
Did you upgrade from a previous version? DId you make some schema changes like compaction strategy, compression, bloom filter, etc.?What about the R/W requests?  SharedPool Workers are... shared ;-) Put logs in debug to see some examples of what services are using this pool (many actually). Bes

Re: CPU high load

2016-07-13 Thread Aoi Kadoya
Hi Patrick, In fact I couldn't see any thread pool named "shared". here is the result of tpstats from one of my nodes. Pool NameActive Pending Completed Blocked All time blocked MutationStage 0 0 173237609 0 0

Re: CPU high load

2016-07-13 Thread Patrick McFadin
Might be more clear looking at nodetool tpstats >From there you can see all the thread pools and if there are any blocks. Could be something subtle like network. On Tue, Jul 12, 2016 at 3:23 PM, Aoi Kadoya wrote: > Hi, > > I am running 6 nodes vnode cluster with DSE 4.8.1, and since few weeks >

CPU high load

2016-07-12 Thread Aoi Kadoya
Hi, I am running 6 nodes vnode cluster with DSE 4.8.1, and since few weeks ago, all of the cluster nodes are hitting avg. 15-20 cpu load. These nodes are running on VMs(VMware vSphere) that have 8vcpu (1core/socket)-16 vRAM.(JVM options : -Xms8G -Xmx8G -Xmn800M) At first I thought this is because