Solr heap memory
Hi All We are using solr cloud 8.8.1. We are frequently seeing memory and CPU reaching 100%. Can someone please suggest ideal way to define java heap memory. Server memory is 62Gb, swap-7Gb Currently we have defined as below in solr in.sh SOLR_HEAP= 46000m SOLR_JAVA_MEMORY= Xms:1024m, Xmx: 7098m Please suggest the way to fine tune the above parameters to avoid memory utilisation. Thanks Hari
Re: Solr heap memory
Try setting you xms and xmx to the same value. You have enough memory to go pretty high so I’d try 31gb, 31000mg. Do not go over 31. Also disable swap on the server Let me know if it helps > On Aug 29, 2021, at 3:56 AM, HariBabu kuruva > wrote: > > Hi All > > We are using solr cloud 8.8.1. > > We are frequently seeing memory and CPU reaching 100%. > > Can someone please suggest ideal way to define java heap memory. > > Server memory is 62Gb, swap-7Gb > > Currently we have defined as below in solr in.sh > > SOLR_HEAP= 46000m > SOLR_JAVA_MEMORY= Xms:1024m, Xmx: 7098m > > Please suggest the way to fine tune the above parameters to avoid memory > utilisation. > > Thanks > Hari
Re: Solr heap memory
Thanks for your reply Dave. Is it required to define both the parameters SOLR_HEAP and SOLR_JAVA_MEM. or can i comment SOLR_HEAP and only define SOLR_JAVA_MEM. Also what highest value of Xmx value i can go if i receive OOM with 31gb. I have only solr running on that node. And could you please let me know the reason to disable swap memory. On Sun, Aug 29, 2021, 1:39 PM Dave wrote: > Try setting you xms and xmx to the same value. You have enough memory to > go pretty high so I’d try 31gb, 31000mg. Do not go over 31. Also disable > swap on the server > > Let me know if it helps > > > On Aug 29, 2021, at 3:56 AM, HariBabu kuruva > wrote: > > > > Hi All > > > > We are using solr cloud 8.8.1. > > > > We are frequently seeing memory and CPU reaching 100%. > > > > Can someone please suggest ideal way to define java heap memory. > > > > Server memory is 62Gb, swap-7Gb > > > > Currently we have defined as below in solr in.sh > > > > SOLR_HEAP= 46000m > > SOLR_JAVA_MEMORY= Xms:1024m, Xmx: 7098m > > > > Please suggest the way to fine tune the above parameters to avoid memory > > utilisation. > > > > Thanks > > Hari >
Re: Solr heap memory
On 8/29/2021 2:38 AM, HariBabu kuruva wrote: Is it required to define both the parameters SOLR_HEAP and SOLR_JAVA_MEM. or can i comment SOLR_HEAP and only define SOLR_JAVA_MEM. Also what highest value of Xmx value i can go if i receive OOM with 31gb. I have only solr running on that node. If both are defined, I do not know which one will actually take effect. Figuring that out would require looking at the startup script and doing some experiments to see what Java actually does. I would personally remove SOLR_JAVA_MEM and only go with SOLR_HEAP. Then you can do something very simple like the following, and the Solr startup script will set both -Xms and -Xmx java options to that value: SOLR_HEAP=4g And could you please let me know the reason to disable swap memory. If a system starts actively swapping, its performance in general will be extremely low. If that happens, it is an indication that there is not enough physical memory and the system needs more, or that configurations need to be adjusted to require less memory. Disabling swap makes it impossible for the OS to try and use disk space as memory. In situations where programs are asking for too much memory and you have swap completely disabled, either Java or the OS will simply kill the process that's asking for too much memory, rather than letting it run and destroy overall performance. --- Responding to something in the OP: It is completely normal to see 100 percent memory utilization on just about any server, whether it's running Solr or not. The OS will use all available memory for caching purposes, to speed everything up. The only time you won't see 100 percent memory usage is when you have far more memory than the system actually needs. For instance, if you had 512GB of memory on a system that only handles megabytes of data. https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems (disclaimer: I wrote the wiki page linked here. Any errors are mine.) Thanks, Shawn
Re: Solr heap memory
Yes. Don’t set those memory restrictions, just xms and xmx, both to 31 gigs. Java has problems past that line and will make the gc go into a bad loop. I can send you a link as to why https://community.datastax.com/questions/3661/why-is-a-32-gb-heap-allocation-not-recommended.html But this is almost like a protected secret > On Aug 29, 2021, at 1:52 PM, Shawn Heisey wrote: > > On 8/29/2021 2:38 AM, HariBabu kuruva wrote: >> Is it required to define both the parameters SOLR_HEAP and SOLR_JAVA_MEM. >> or can i comment SOLR_HEAP and only define SOLR_JAVA_MEM. >> Also what highest value of Xmx value i can go if i receive OOM with 31gb. >> I have only solr running on that node. > > If both are defined, I do not know which one will actually take effect. > Figuring that out would require looking at the startup script and doing some > experiments to see what Java actually does. > > I would personally remove SOLR_JAVA_MEM and only go with SOLR_HEAP. Then you > can do something very simple like the following, and the Solr startup script > will set both -Xms and -Xmx java options to that value: > > SOLR_HEAP=4g > >> And could you please let me know the reason to disable swap memory. > > If a system starts actively swapping, its performance in general will be > extremely low. If that happens, it is an indication that there is not enough > physical memory and the system needs more, or that configurations need to be > adjusted to require less memory. > > Disabling swap makes it impossible for the OS to try and use disk space as > memory. In situations where programs are asking for too much memory and you > have swap completely disabled, either Java or the OS will simply kill the > process that's asking for too much memory, rather than letting it run and > destroy overall performance. > > --- > > Responding to something in the OP: > > It is completely normal to see 100 percent memory utilization on just about > any server, whether it's running Solr or not. The OS will use all available > memory for caching purposes, to speed everything up. The only time you won't > see 100 percent memory usage is when you have far more memory than the system > actually needs. For instance, if you had 512GB of memory on a system that > only handles megabytes of data. > > https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems > > (disclaimer: I wrote the wiki page linked here. Any errors are mine.) > > Thanks, > Shawn
Sort results based on alpha at first
Hi, We need to sort the results based on alphabetic followed by numeric. Something like below Current results order: 1test 2test 3test Abctext Bbctext Cbctext Expected results order: Abctext Bbctext Cbctext 1test 2test 3test Please help me to achieve the result ordering as above. Thanks, Thamizh -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus
Re: Sort results based on alpha at first
On 8/29/2021 12:09 PM, post.ta...@gmail.com wrote: We need to sort the results based on alphabetic followed by numeric. Something like below Expected results order: Abctext Bbctext Cbctext 1test 2test 3test This is contrary to all the "natural" sort ordering available in modern programming languages. In the ASCII table, numbers come before letters. Though one may exist, I have never heard about anything to do this in Java. You would have to write (or find) custom sorting code to achieve this. And then somehow get it integrated into Solr ... and I have no idea how to integrate custom sorting algorithms. Alternatively, you could return more results than the user actually asks for, sorted some other way like relevancy score, and then do the sort yourself in the client code (like a webapp). Thanks, Shawn
Re: Sort results based on alpha at first
Hi Thamizh You can add a field to the index that acts as a "sortable flag" with e.g. 0 for alpha, 1 for numerical, and 2 for other first character. Then you can sort on both fields, listing the "flag" field first. Thomas Op zo 29 aug. 2021 20:09 schreef : > Hi, > > > > We need to sort the results based on alphabetic followed by numeric. > Something like below > > > > Current results order: > > 1test > > 2test > > 3test > > Abctext > > Bbctext > > Cbctext > > > > Expected results order: > > Abctext > > Bbctext > > Cbctext > > 1test > > 2test > > 3test > > > > Please help me to achieve the result ordering as above. > > > > Thanks, > > Thamizh > > > > -- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus >
