Thanks Shawn, I did as you suggested and replaced Xmx/Xms (I'm on FreeBSD by the way) with SOLR_HEAP='16g' and I see it now uses that value instead for the running process.
/usr/local/openjdk11/bin/java -server -Xms16g -Xmx16g -XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:+ExplicitGCInvokesConcurrent However, top still shows Solr using more than 16g . It started at 17g and has been steadily growing, now it's at 23g and soon it will go into swap PID USERNAME THR PRI NICE SIZE RES SWAP STATE C TIME WCPU COMMAND 57116 solr 165 52 0 235G 23G 0 uwait 1 36:20 9.60% java Is it normal for the process to occupy in memory more than the configured 16g ? Best, Scott -----Original Message----- From: Shawn Heisey <apa...@elyograg.org> Sent: Saturday, December 11, 2021 1:30 PM To: users@solr.apache.org Subject: Re: Solr Cloud Node re-join issue On 12/11/2021 8:39 AM, Scott wrote: > Thanks Shawn. Most people I talked to sort of acknowledged that restarting > SOLR every month or so is a given, but your comments are encouraging. > > These nodes have 32Gb of ram: > > real memory = 34359738368 (32768 MB) > avail memory = 33370628096 (31824 MB) > > and here's what I have in my solr config > > SOLR_JAVA_MEM="-Xms14512m -Xmx16512m" I have noticed with Linux servers under my care that they do utilize more swap than I would have ever expected. But most of them seem to be healthy despite that. If your machines are running Linux, then the following will apply. For other UNIX-like operating systems, I have no idea whether you can use this info: --- This shell script will show you what programs are utilizing swap. It must be run as root: https://apaste.info/Gnwx It's not actually important whether there is swap in use. What's really important is whether or not the system is ACTIVELY swapping. You can see that with the following command, which I recommend running with a terminal width well beyond 80: vmstat 2 -w If the si and so columns frequently show numbers other than 0, then there is active swapping going on. If it is almost always 0 or a very small number, then there is probably no need to worry about swap usage. --- With the "top" utility on most UNIX-like operating systems, there is a way we can see the overall health of memory usage on a system. If my email program clobbers this link, I apologize: https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue > I thought I'd keep it at half of available RAM but it still goes into swap... I really want to smack the person who came up with the rule of thumb where a Java heap gets assigned to be some fraction of the total memory size, usually between 25 and 50 percent. In practice, that advice usually means that the heap will be too small or significantly larger than it needs to be. What you should do for the heap size is figure out how much heap memory the program actually needs to do its work, then assign the max heap to a number above that which gives the program some headroom to operate in. I do not have any generic guidelines for how much headroom is needed ... it will depend on the exact situation you find yourself in. We also recommend setting the min heap and max heap to the same number. For your system that would mean this (instead of SOLR_JAVA_MEM): SOLR_HEAP="16g" Thanks, Shawn This is a private message
