On 12/12/2021 4:40 PM, Scott wrote:
However, top still shows Solr using more than 16g . It started at 17g and has
been steadily growing, now it's at 23g and soon it will go into swap
PID USERNAME THR PRI NICE SIZE RES SWAP STATE C TIME
WCPU COMMAND
57116 solr 165 52 0 235G 23G 0 uwait
1 36:20 9.60% java
I have been doing some experiments with a FreeBSD VM running in vmware
player. I have very little experience with that OS.
I have openjdk11 and apache-solr-8.10.0 installed using pkg.
It looks like top in FreeBSD has no equivalent to the SHR column seen on
Linux. Being able to see how much shared memory is being used is
critical to seeing a complete picture of memory usage. I suspect that
if we could see it, the shared memory would be approximately 7GB when
the RES column says 23GB. This is something I have seen on Linux, and I
have deduced that the actual memory used by the process will be the RES
size minus the SHR size. Sometimes the shared memory will get quite
large. I have no idea why this happens, but it does.
There is a Java tool called "jstat" that can give a very accurate
picture of Java program memory usage. But when the
-XX:+PerfDisableSharedMem option is given to Java, that tool doesn't
work. That option is added with the default GC tuning options, because
it eliminates a severe performance issue that is sometimes seen with
Java software.
If you add the following to solr.in.sh then jstat will work:
GC_TUNE="-XX:+UseG1GC \
-XX:+ParallelRefProcEnabled \
-XX:MaxGCPauseMillis=250 \
-XX:+UseLargePages \
-XX:+AlwaysPreTouch \
-XX:+ExplicitGCInvokesConcurrent"
After adding it, restart Solr and use the following command with the PID
of the Solr process in place of PID, at a time when the RES column for
Solr goes well beyond 16GB:
sudo jstat -gc -t PID 5000 20 > /tmp/jstat.out
That command will take a little less than two minutes to complete. Then
you can share the /tmp/jstat.out file using a file sharing website.
Don't try to paste it into email ... the lines are VERY long.
If you add up the columns named S0C, S1C, EC, OC, MC, and CCSC for a
given line, that will be pretty close to the process's total memory
usage, in KB.
If you want to know what all those columns mean, here's Oracle's
documentation for jstat:
https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html
I've just gotten a look at the output from vmstat ... looks like that
tool is useless for what I was trying to get from it. It doesn't have
any columns for swap. You may have noticed that the si and so columns I
mentioned before are not present.
It is worth noting that on the top output you pasted, that the Solr
process is using zero swap. On the top screen, are there processes with
SWAP columns significantly larger than zero? When you see a problem,
what is the output of "swapinfo" ?
Thanks,
Shawn
