James,
On 10/27/20 16:20, James H. H. Lampert wrote:
This is related to my query (thanks, Mr. Gregg) about "Tenured SOA."
It seems that on one of our customer installations, our webapp gets into
a state of running very slowly, and the dedicated subsystem it's running
in is showing massive levels of page-faulting.
I've compared the GC stats of the "problem" system with one that's
actually got more users connected, but isn't experiencing performance
issues. It seems that they're both going to GC about every 30-50
seconds, but GC on the "problem" system appears to be somewhat less
effective.
Also, I've looked at the threads on both. On the system that is behaving
normally, the "GC Slave" threads (7 of them) are showing total CPU (at
this moment) of around 150 seconds each, and Aux I/O of mostly zero,
with one showing 1 and one showing 3. Conversely, on the "problem"
system, I'm seeing 15(!) GC Slave threads, each with total CPU under 6
seconds each, but Aux I/O ranging from 5800 to over 8000.
I'm not sure what to make of this. In both cases, Tomcat's JVM is
running in a subsystem of its own, with a private memory pool of around
7G, and they're both running with -Xms4096m -Xmx5120m.
If you expect the service to be long-running, definitely set Xms=Xmx.
There's no reason to artificially restrict the heap "early" in the
process's lifetime only to completely re-size and re-organize the heap
over time. You may as well allocate the maximum right up front and leave
it that way.
The problem system certainly appears to be thrashing its GC. Are there
any environmental differences that you notice about the two systems? For
a JVM with a maximum heap of ~5GiB, I think that a 7GiB private memory
space (this is an AS/400 thing isn't it?) isn't large enough. The heap
space is just the "Java heap" and there are other things that need
memory, sometimes ~= to the heap size. It's sometimes surprising how
much "native" memory a JVM needs. Is the kernel+userspace running in
that "subsystem" as well? Or just the JVM process?
I'm guessing that your comment about page-faulting and "Aux I/O rang[es]
from 5800 - 8000 [sec]" means that you are actually paging the heap to
the disk. What happens if you shrink your max-heap to 2GiB and change
nothing else? This should make sure that your heap + native memory fits
into physical memory and that thrashing should stop.
Maybe you *do* need a 5GiB heap, though. In that case, if the
heap-shrink works but you get OOMEs under load, then I think that simply
increasing the memory allocated to the "subsystem" should help a lot.
How much (real) memory does the system report is being used by the JVM
process? I think you'll find it much larger than 5GiB.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
