Good to know. So that confirms it's just the GC threads causing problems. On Tue., 29 May 2018, 22:02 Steinmaurer, Thomas, < thomas.steinmau...@dynatrace.com> wrote:
> Kurt, > > > > in our test it also didn’t made a difference with the default number of GC > Threads (43 on our large machine) and running with Xmx128M or XmX31G > (derived from $MAX_HEAP_SIZE). For both Xmx, we saw the high CPU caused by > nodetool. > > > > Regards, > > Thomas > > > > *From:* kurt greaves [mailto:k...@instaclustr.com] > *Sent:* Dienstag, 29. Mai 2018 13:06 > *To:* User <user@cassandra.apache.org> > *Subject:* Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage > > > > Thanks Thomas. After a bit more research today I found that the whole > $MAX_HEAP_SIZE issue isn't really a problem because we don't explicitly set > -Xms so the minimum heapsize by default will be 256mb, which isn't hugely > problematic, and it's unlikely more than that would get allocated. > > > > On 29 May 2018 at 09:29, Steinmaurer, Thomas < > thomas.steinmau...@dynatrace.com> wrote: > > Hi Kurt, > > > > thanks for pointing me to the Xmx issue. > > > > JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread > issue is available here: > https://issues.apache.org/jira/browse/CASSANDRA-14475 > > > > Thanks, > > Thomas > > > > *From:* kurt greaves [mailto:k...@instaclustr.com] > *Sent:* Dienstag, 29. Mai 2018 05:54 > *To:* User <user@cassandra.apache.org> > *Subject:* Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage > > > > 1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we > are running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as > well > > This was fixed in 3.0.11/3.10 in CASSANDRA-12739 > <https://issues.apache.org/jira/browse/CASSANDRA-12739>. Not sure why it > didn't make it into 2.1/2.2. > > 2) As -XX:ParallelGCThreads is not explicitly set upon startup, this > basically defaults to a value dependent on the number of cores. In our > case, with the machine above, the number of parallel GC threads for the JVM > is set to 43! > 3) Test-wise, we have adapted the nodetool startup script in a way to get > a Java Flight Recording file on JVM exit, thus with each nodetool > invocation we can inspect a JFR file. Here we may have seen System.gc() > calls (without visible knowledge where they come from), GC times for the > entire JVM life-time (e.g. ~1min) showing high cpu. This happened for both > Xmx128M (default as it seems) and Xmx31G > > After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup > script, CPU usage spikes by nodetool are entirely gone. > > Is this something which has been already adapted/tackled in Cassandra > versions > 2.1 or worth to be considered as some sort of RFC? > > Can you create a JIRA for this (and a patch, if you like)? We should be > explicitly setting this on nodetool invocations. > > > > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or > disclose it to anyone else. If you received it in error please notify us > immediately and then destroy it. Dynatrace Austria GmbH (registration > number FN 91482h) is a company registered in Linz whose registered office > is at 4040 Linz, Austria, Freistädterstraße 313 > > > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or > disclose it to anyone else. If you received it in error please notify us > immediately and then destroy it. Dynatrace Austria GmbH (registration > number FN 91482h) is a company registered in Linz whose registered office > is at 4040 Linz, Austria, Freistädterstraße 313 >
