So, just as a follow up, I'm going to post my BEFORE and AFTER solr.in.sh configs in case I've removed anything vital. Any comments welcome, as they don't really mean much to me. === BEFORE ==== SOLR_HEAP="8000m"
GC_LOG_OPTS="-verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails \ -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime" GC_TUNE="-XX:NewRatio=3 \ -XX:SurvivorRatio=4 \ -XX:TargetSurvivorRatio=90 \ -XX:MaxTenuringThreshold=8 \ -XX:+UseConcMarkSweepGC \ -XX:+UseParNewGC \ -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \ -XX:+CMSScavengeBeforeRemark \ -XX:PretenureSizeThreshold=64m \ -XX:+UseCMSInitiatingOccupancyOnly \ -XX:CMSInitiatingOccupancyFraction=50 \ -XX:CMSMaxAbortablePrecleanTime=6000 \ -XX:+CMSParallelRemarkEnabled \ -XX:+ParallelRefProcEnabled" ENABLE_REMOTE_JMX_OPTS="false" SOLR_OPTS="$SOLR_OPTS -Xss256k" SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true" =============== ====AFTER===== SOLR_HEAP="8000m" GC_LOG_OPTS="-verbose:gc -Xlog:gc*" GC_TUNE="-XX:NewRatio=3 \ -XX:SurvivorRatio=4 \ -XX:TargetSurvivorRatio=90 \ -XX:MaxTenuringThreshold=8 \ -XX:+UseG1GC \ -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \ -XX:+CMSScavengeBeforeRemark \ -XX:PretenureSizeThreshold=64m \ -XX:+UseCMSInitiatingOccupancyOnly \ -XX:CMSInitiatingOccupancyFraction=50 \ -XX:CMSMaxAbortablePrecleanTime=6000 \ -XX:+CMSParallelRemarkEnabled \ -XX:+ParallelRefProcEnabled" ENABLE_REMOTE_JMX_OPTS="false" SOLR_OPTS="$SOLR_OPTS -Xss256k" SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true" =============== On Wed, Dec 13, 2023 at 4:09 PM Jim Morgan <jim.mor...@nknews.org> wrote: > I've had some success. Firstly, Shawn I did find update-java-alternatives > with my googling and what an eye-opener that was! I thought it would surely > solve the problem, but it unfortunately didn't. > > So today, I tried again. Because I hadn't noticed anything in the solr > logs before, I decided to wipe the entire log directory, and see what > appeared. I found my clue in the previously unnoticed solr-8983-console.log > I'd been looking in solr.log and the various gc logs, but hadn't noticed > this one appearing. In fact it was the only one which was created. This > mentioned a deprecated setting in my solr.in.sh which I removed, ran > again, removed etc until it finally ran, waiting for the mandatory 180 > seconds in between each restart. > These are the things I removed before it worked > -XX:+PrintGCDetails > UseConcMarkSweepGC > PrintGCApplicationStoppedTime > PrintGCDateStamps > PrintGCTimeStamps > PrintHeapAtGC > PrintTenuringDistribution > UseParNewGC > > So now it all seems to be chugging away on Java 11. Will probably leave it > on that for a while, until I have an urge to tangle with solr again. > Understand that java 13 is a no-no, as its not LTS. > > Thanks for your suggestions and advice. > > Jim > > On Wed, Dec 13, 2023 at 2:06 AM Chris Hostetter <hossman_luc...@fucit.org> > wrote: > >> >> That's the console output, where the start script is waiting for the port >> to come up. >> >> what does the actual solr log file (written by the java process) say? >> >> >> https://solr.apache.org/guide/solr/latest/deployment-guide/taking-solr-to-production.html#log-settings >> >> : Date: Tue, 12 Dec 2023 10:57:37 +0800 >> : From: Jim Morgan <jim.mor...@nknews.org.invalid> >> : Reply-To: users@solr.apache.org >> : To: users@solr.apache.org >> : Subject: Re: Java Upgrade process >> : >> : Fair point Chris, I should have included log info in the original post. >> : So in the solr logs, nothing at all. I think because solr doesn't get to >> : the point of starting. >> : In syslog I have >> : --------- >> : Dec 11 06:35:08 server solr[6822]: Waiting up to 180 seconds to see Solr >> : running on port 8983 [|] #010#010#010#010#010#010 [/] >> : #010#010#010#010#010#010 [|] #010#010#010#010#010#010 [/] >> : (lots and lots of these removed) >> : 010#010#010 [-] #010#010#010#010#010 >> : Dec 11 06:35:08 server solr[6897]: Still not seeing Solr listening on >> 8983 >> : after 180 seconds! >> : --------- >> : There was a warning about Ulimits, which doesn't seem to occur under >> Java 8: >> : Dec 11 06:30:32 solr-dev solr[6266]: *** [WARN] *** Your open file >> limit is >> : currently 1024. >> : Dec 11 06:30:32 solr-dev solr[6266]: It should be set to 65000 to avoid >> : operational disruption. >> : Dec 11 06:30:32 solr-dev solr[6266]: If you no longer wish to see this >> : warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh >> : But I set that in /etc/defaults/solr.in.sh and it didn't help. >> : >> : Jim >> : >> : On Tue, Dec 12, 2023 at 8:10 AM Chris Hostetter < >> hossman_luc...@fucit.org> >> : wrote: >> : >> : > >> : > can you be more specific about " It just waits and eventually fails >> after >> : > about 2 minutes." ... what message does it print to the consolr? what >> : > does the solr log file say? >> : > >> : > >> : > >> : > : Date: Mon, 11 Dec 2023 11:36:15 +0800 >> : > : From: Jim Morgan <jim.mor...@nknews.org.invalid> >> : > : Reply-To: users@solr.apache.org >> : > : To: users@solr.apache.org >> : > : Subject: Java Upgrade process >> : > : >> : > : A few months back, I had a question about upgrading Solr. I was >> trying to >> : > : upgrade from 8.5 to 9.3. That introduced a few errors into our >> website >> : > : search that we were not able to resolve, so we've figured we'll just >> : > : upgrade to the latest 8.x version, ie. 8.11. >> : > : >> : > : We've successfully upgraded the underlying OS, and we're happy 8.11 >> : > returns >> : > : the same search results as 8.5. So far so good. >> : > : >> : > : Now we're looking at java. Its currently running on openjdk-8. I've >> tried >> : > : upgrading to openjdk-11 and openjdk-13. The process, as this is >> Ubuntu is >> : > : basically. >> : > : apt install openjdk-13-jdk >> : > : update-alternatives --list java >> : > : update-alternatives --set java >> : > /usr/lib/jvm/java-13-openjdk-amd64/bin/java >> : > : On doing that, solr will fail to start. It just waits and eventually >> : > fails >> : > : after about 2 minutes. So I can do >> : > : update-alternatives --set java >> : > : /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java >> : > : To set it back to java 8 and it runs again. >> : > : >> : > : I've tried regenerating the init script with the new java, but the >> : > scripts >> : > : are identical. I've tried rebooting the server with the new java >> active >> : > in >> : > : case its an environment thing. But to no avail. >> : > : >> : > : Any ideas what's going on here? >> : > : >> : > : I notice java 8 is still getting updates, so maybe the path of least >> : > : resistance is to stay on that. >> : > : >> : > : Jim >> : > : >> : > >> : > -Hoss >> : > http://www.lucidworks.com/ >> : > >> : >> >> -Hoss >> http://www.lucidworks.com/ > >
