Starting Solr 8 with Java 8 and SSL

[Horton, Alan]

I have updated a Solr install to Solr 8.10 from 7.7.3, this works with no 
issues using http, but as soon as I try to use https it will not start.

I have tried adding  -Dsolr.http1=true  to the command line but with no joy.

Is there some other configuration I can apply to force Solr into using HTTP/1.1 
rather than 2 ?

Thanks

ALAN HORTON
Senior DevOps Engineer
Financial Messaging Marketplaces

[FINASTRA]

Unit E, Nottingham One, 154 Canal Street, Nottingham, NG1 7HG
Email: alan.hor...@finastra.com
Finastra.com

"FINASTRA" is the trade name of the FINASTRA group of companies. This email and 
any attachments have been scanned for known viruses using multiple scanners. 
This email message is intended for the named recipient only. It may be 
privileged and/or confidential. If you are not the named recipient of this 
email please notify us immediately and do not copy it or use it for any 
purpose, nor disclose its contents to any other person. This email does not 
constitute the commencement of legal relations between you and FINASTRA. Please 
refer to the executed contract between you and the relevant member of the 
FINASTRA group for the identity of the contracting party with which you are 
dealing.

Re: Starting Solr 8 with Java 8 and SSL

[Gus Heck]

What error message do you get when you try to start solr with https? Can
you provide more detail as to what you did to "use https"? Hard to know if
switching to 1.1 would be a good idea here until we understand the problem
better.

On Mon, Dec 13, 2021 at 6:18 AM Horton, Alan
 wrote:

> I have updated a Solr install to Solr 8.10 from 7.7.3, this works with no
> issues using http, but as soon as I try to use https it will not start.
>
>
>
> I have tried adding  -Dsolr.http1=true  to the command line but with no
> joy.
>
>
>
> Is there some other configuration I can apply to force Solr into using
> HTTP/1.1 rather than 2 ?
>
>
>
> Thanks
>
>
>
> *ALAN HORTON*
>
> Senior DevOps Engineer
>
> Financial Messaging Marketplaces
>
>
>
> [image: FINASTRA]
>
>
>
> Unit E, Nottingham One, 154 Canal Street, Nottingham, NG1 7HG
>
> *Email: *alan.hor...@finastra.com
>
> *Finastra.com
> *
>
>
> "FINASTRA" is the trade name of the FINASTRA group of companies. This
> email and any attachments have been scanned for known viruses using
> multiple scanners. This email message is intended for the named recipient
> only. It may be privileged and/or confidential. If you are not the named
> recipient of this email please notify us immediately and do not copy it or
> use it for any purpose, nor disclose its contents to any other person. This
> email does not constitute the commencement of legal relations between you
> and FINASTRA. Please refer to the executed contract between you and the
> relevant member of the FINASTRA group for the identity of the contracting
> party with which you are dealing.
>


-- 
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Zookeeper and Solr and CVE-2021-44228

[Michael Conrad]

I presume this also needs fixing for zookeeper nodes?

On 12/10/21 13:44, Walter Underwood wrote:

Does all Solr logging go through slf4j? If so, that should protect against this 
vulnerability.

If not, who has tested Solr with log4j 2.15.1?

We are running 8.8.2.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/   (my blog)

RE: [EXT] Re: Starting Solr 8 with Java 8 and SSL

[Horton, Alan]

Initially a series of sessionExpiredExceptions:

Eventually a timeoutException:


Thanks


-Original Message-
From: Gus Heck mailto:gus.h...@gmail.com>>
Sent: 13 December 2021 14:10
To: users@solr.apache.org
Subject: [EXT] Re: Starting Solr 8 with Java 8 and SSL

What error message do you get when you try to start solr with https? Can
you provide more detail as to what you did to "use https"? Hard to know if
switching to 1.1 would be a good idea here until we understand the problem
better.

On Mon, Dec 13, 2021 at 6:18 AM Horton, Alan
mailto:alan.hor...@finastra.com.invalid>> 
wrote:

> I have updated a Solr install to Solr 8.10 from 7.7.3, this works with no
> issues using http, but as soon as I try to use https it will not start.
>
>
>
> I have tried adding  -Dsolr.http1=true  to the command line but with no
> joy.
>
>
>
> Is there some other configuration I can apply to force Solr into using
> HTTP/1.1 rather than 2 ?
>
>
>
> Thanks
>
>
>
> *ALAN HORTON*
>
> Senior DevOps Engineer
>
> Financial Messaging Marketplaces
>
>
>
> [image: FINASTRA]
>
>
>
> Unit E, Nottingham One, 154 Canal Street, Nottingham, NG1 7HG
>
> *Email: *alan.hor...@finastra.com
>
> *Finastra.com
> *
>
>
> "FINASTRA" is the trade name of the FINASTRA group of companies. This
> email and any attachments have been scanned for known viruses using
> multiple scanners. This email message is intended for the named recipient
> only. It may be privileged and/or confidential. If you are not the named
> recipient of this email please notify us immediately and do not copy it or
> use it for any purpose, nor disclose its contents to any other person. This
> email does not constitute the commencement of legal relations between you
> and FINASTRA. Please refer to the executed contract between you and the
> relevant member of the FINASTRA group for the identity of the contracting
> party with which you are dealing.
>


--
https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.needhamsoftware.com%2F&data=04%7C01%7CAlan.Horton%40finastra.com%7Cb8e36427094c4056a18708d9be424a0a%7C0b9b90da3fe1457ab340f1b67e1024fb%7C0%7C0%7C637750015038762234%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xCaIB1k2MZwZA5Jl3ZPJEj7AKXhTiKdGxZGTY2i4HOc%3D&reserved=0
 (work)
https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.the111shift.com%2F&data=04%7C01%7CAlan.Horton%40finastra.com%7Cb8e36427094c4056a18708d9be424a0a%7C0b9b90da3fe1457ab340f1b67e1024fb%7C0%7C0%7C637750015038762234%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=4De%2F6iZ95hmcTj2dwu7OovM3nYyqoyLkbEbjyegbyEM%3D&reserved=0
 (play)

"FINASTRA" is the trade name of the FINASTRA group of companies. This email and 
any attachments have been scanned for known viruses using multiple scanners. 
This email message is intended for the named recipient only. It may be 
privileged and/or confidential. If you are not the named recipient of this 
email please notify us immediately and do not copy it or use it for any 
purpose, nor disclose its contents to any other person. This email does not 
constitute the commencement of legal relations between you and FINASTRA. Please 
refer to the executed contract between you and the relevant member of the 
FINASTRA group for the identity of the contracting party with which you are 
dealing.

Re: Zookeeper and Solr and CVE-2021-44228

[Andy Lester]

> On Dec 13, 2021, at 8:20 AM, Michael Conrad  wrote:
> 
> I presume this also needs fixing for zookeeper nodes?

Anything that logs with log4j.

Re: Zookeeper and Solr and CVE-2021-44228

[Andy C]

Zookeeper has not yet migrated to log4j2. Even their latest releases
(3.6.3, 3.7.0) are still using version 1.2.17 of log4j.

So I would think that Zookeeper would be in the same situation as the
pre-7.4.0 Solr releases as described here:
https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228

So I guess the question is whether Zookeeper uses the JMS Appender?

- Andy -

On Mon, Dec 13, 2021 at 9:30 AM Andy Lester  wrote:

>
>
> > On Dec 13, 2021, at 8:20 AM, Michael Conrad  wrote:
> >
> > I presume this also needs fixing for zookeeper nodes?
>
> Anything that logs with log4j.

RE: Solr Cloud Node re-join issue

[Scott]

So you're saying that Solr can use additional memory on top of what Xmx limits 
? It appears the resident size keeps on increasing, swap was used at some point 
but it's not actively paging now.

In FreeBSD you can use procstat or vmstat for more info

root@solrcloud4:/usr/home/scott # procstat -r 57116
  PID COMM RESOURCE  VALUE
57116 java user time07:59:00.567533
57116 java system time  00:31:36.921875
57116 java maximum RSS 31175620 KB
57116 java integral shared memory 139301820 KB
57116 java integral unshared data  46433940 KB
57116 java integral unshared stack495295360 KB
57116 java page reclaims   11361433
57116 java page faults  2919735
57116 java swaps  0
57116 java block reads  2918179
57116 java block writes 1724654
57116 java messages sent   23464275
57116 java messages received   13276352
57116 java signals received   28619
57116 java voluntary context switches  66989710
57116 java involuntary context switches49835294

So with a SOLR_HEAP value of 16g , I'm now at a total size of 30g which has 
already used some swap that it hasn't released yet. This will keep on going 
until swap usage is 100% and the box will crash.

I guess my questions are:
- Why does Solr use more than 16g ?
- Why isn't swapped memory released ?

Thanks!
Scott

-Original Message-
From: Shawn Heisey  
Sent: Monday, December 13, 2021 12:41 AM
To: users@solr.apache.org
Subject: Re: Solr Cloud Node re-join issue

On 12/12/2021 4:40 PM, Scott wrote:
> However, top still shows Solr using more than 16g . It started at 17g 
> and has been steadily growing, now it's at 23g and soon it will go 
> into swap
> 
> PID  USERNAMETHR PRI NICE   SIZE  RES  SWAP STATEC   TIME
> WCPU COMMAND
> 57116 solr  165  520   235G23G0   uwait   
>  1  36:20   9.60% java

I have been doing some experiments with a FreeBSD VM running in vmware player.  
I have very little experience with that OS.

I have openjdk11 and apache-solr-8.10.0 installed using pkg.

It looks like top in FreeBSD has no equivalent to the SHR column seen on Linux. 
 Being able to see how much shared memory is being used is critical to seeing a 
complete picture of memory usage.  I suspect that if we could see it, the 
shared memory would be approximately 7GB when the RES column says 23GB.  This 
is something I have seen on Linux, and I have deduced that the actual memory 
used by the process will be the RES size minus the SHR size.  Sometimes the 
shared memory will get quite large.  I have no idea why this happens, but it 
does.

There is a Java tool called "jstat" that can give a very accurate picture of 
Java program memory usage.  But when the -XX:+PerfDisableSharedMem option is 
given to Java, that tool doesn't work.  That option is added with the default 
GC tuning options, because it eliminates a severe performance issue that is 
sometimes seen with Java software.

If you add the following to solr.in.sh then jstat will work:

GC_TUNE="-XX:+UseG1GC \
   -XX:+ParallelRefProcEnabled \
   -XX:MaxGCPauseMillis=250 \
   -XX:+UseLargePages \
   -XX:+AlwaysPreTouch \
   -XX:+ExplicitGCInvokesConcurrent"

After adding it, restart Solr and use the following command with the PID of the 
Solr process in place of PID, at a time when the RES column for Solr goes well 
beyond 16GB:

sudo jstat -gc -t PID 5000 20 > /tmp/jstat.out

That command will take a little less than two minutes to complete.  Then you 
can share the /tmp/jstat.out file using a file sharing website. 
Don't try to paste it into email ... the lines are VERY long.

If you add up the columns named S0C, S1C, EC, OC, MC, and CCSC for a given 
line, that will be pretty close to the process's total memory usage, in KB.

If you want to know what all those columns mean, here's Oracle's documentation 
for jstat:

https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html

I've just gotten a look at the output from vmstat ... looks like that tool is 
useless for what I was trying to get from it.  It doesn't have any columns for 
swap.  You may have noticed that the si and so columns I mentioned before are 
not present.

It is worth noting that on the top output you pasted, that the Solr process is 
using zero swap.  On the top screen, are there processes with SWAP columns 
significantly larger than zero?  When you see a problem, what is the output of 
"swapinfo" ?

Thanks,
Shawn



This is a private message

Re: Zookeeper and Solr and CVE-2021-44228

[Walter Underwood]

Zookeeper 3.5.7 uses log4j 1.x, so is not vulnerable. I checked.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Dec 13, 2021, at 6:20 AM, Michael Conrad  wrote:
> 
> I presume this also needs fixing for zookeeper nodes?
> 
> On 12/10/21 13:44, Walter Underwood wrote:
>> Does all Solr logging go through slf4j? If so, that should protect against 
>> this vulnerability.
>> 
>> If not, who has tested Solr with log4j 2.15.1?
>> 
>> We are running 8.8.2.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/   (my blog)
>> 
>> 

Re: [ANNOUNCEMENT] Solr's Docker images were updated to remediate a CVE

[Andy Lester]

For those of you like me who want to explicitly set the variable without 
relying on which of the two Docker images with the same tag you’re pulling 
down, and you’re using a Dockerfile to add on to make your own Solr Docker 
image, add these lines:

# Add option to mitigate log4j security vulnerability.
USER root
RUN echo 'SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true"' >> 
/etc/default/solr.in.sh
USER solr

Andy

Re: [EXT] Re: Starting Solr 8 with Java 8 and SSL

[Robert Pearce]

This threw me for a while when I worked on it.
Visit this page 
https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=55153140#content/view/55153140
You need to treat Solr as a Zookeeper client and add the six system properties 
it recommends for ziCli.sh to your Solr.in.sh or similar file, e.g. via 
SOLR_OPTS. These will be about the netty socket, ask keystone etc

These aren’t discussed on the Solr Enabling SSL page but without them Solr 
wouldn’t connect to ZK via SSL

Sent from my iPhone

> On 13 Dec 2021, at 14:27, Horton, Alan  
> wrote:
> 
> Initially a series of sessionExpiredExceptions:
> 
> Eventually a timeoutException:
> 
>  
> Thanks
>  
>  
> -Original Message-
> From: Gus Heck  
> Sent: 13 December 2021 14:10
> To: users@solr.apache.org
> Subject: [EXT] Re: Starting Solr 8 with Java 8 and SSL
>  
> What error message do you get when you try to start solr with https? Can
> you provide more detail as to what you did to "use https"? Hard to know if
> switching to 1.1 would be a good idea here until we understand the problem
> better.
>  
> On Mon, Dec 13, 2021 at 6:18 AM Horton, Alan
>  wrote:
>  
> > I have updated a Solr install to Solr 8.10 from 7.7.3, this works with no
> > issues using http, but as soon as I try to use https it will not start.
> >
> >
> >
> > I have tried adding  -Dsolr.http1=true  to the command line but with no
> > joy.
> >
> >
> >
> > Is there some other configuration I can apply to force Solr into using
> > HTTP/1.1 rather than 2 ?
> >
> >
> >
> > Thanks
> >
> >
> >
> > *ALAN HORTON*
> >
> > Senior DevOps Engineer
> >
> > Financial Messaging Marketplaces
> >
> >
> >
> > [image: FINASTRA]
> >
> >
> >
> > Unit E, Nottingham One, 154 Canal Street, Nottingham, NG1 7HG
> >
> > *Email: *alan.hor...@finastra.com
> >
> > *Finastra.com
> > *
> >
> >
> > "FINASTRA" is the trade name of the FINASTRA group of companies. This
> > email and any attachments have been scanned for known viruses using
> > multiple scanners. This email message is intended for the named recipient
> > only. It may be privileged and/or confidential. If you are not the named
> > recipient of this email please notify us immediately and do not copy it or
> > use it for any purpose, nor disclose its contents to any other person. This
> > email does not constitute the commencement of legal relations between you
> > and FINASTRA. Please refer to the executed contract between you and the
> > relevant member of the FINASTRA group for the identity of the contracting
> > party with which you are dealing.
> >
>  
>  
> --
> https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.needhamsoftware.com%2F&data=04%7C01%7CAlan.Horton%40finastra.com%7Cb8e36427094c4056a18708d9be424a0a%7C0b9b90da3fe1457ab340f1b67e1024fb%7C0%7C0%7C637750015038762234%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xCaIB1k2MZwZA5Jl3ZPJEj7AKXhTiKdGxZGTY2i4HOc%3D&reserved=0
>  (work)
> https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.the111shift.com%2F&data=04%7C01%7CAlan.Horton%40finastra.com%7Cb8e36427094c4056a18708d9be424a0a%7C0b9b90da3fe1457ab340f1b67e1024fb%7C0%7C0%7C637750015038762234%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=4De%2F6iZ95hmcTj2dwu7OovM3nYyqoyLkbEbjyegbyEM%3D&reserved=0
>  (play)
>  
> "FINASTRA" is the trade name of the FINASTRA group of companies. This email 
> and any attachments have been scanned for known viruses using multiple 
> scanners. This email message is intended for the named recipient only. It may 
> be privileged and/or confidential. If you are not the named recipient of this 
> email please notify us immediately and do not copy it or use it for any 
> purpose, nor disclose its contents to any other person. This email does not 
> constitute the commencement of legal relations between you and FINASTRA. 
> Please refer to the executed contract between you and the relevant member of 
> the FINASTRA group for the identity of the contracting party with which you 
> are dealing.

Re: [EXT] Re: Starting Solr 8 with Java 8 and SSL

[Robert Pearce]

Apologies for any typos, using a phone

> On 13 Dec 2021, at 14:27, Horton, Alan  
> wrote:
> 
> Initially a series of sessionExpiredExceptions:
> 
> Eventually a timeoutException:
> 
>  
> Thanks
>  
>  
> -Original Message-
> From: Gus Heck  
> Sent: 13 December 2021 14:10
> To: users@solr.apache.org
> Subject: [EXT] Re: Starting Solr 8 with Java 8 and SSL
>  
> What error message do you get when you try to start solr with https? Can
> you provide more detail as to what you did to "use https"? Hard to know if
> switching to 1.1 would be a good idea here until we understand the problem
> better.
>  
> On Mon, Dec 13, 2021 at 6:18 AM Horton, Alan
>  wrote:
>  
> > I have updated a Solr install to Solr 8.10 from 7.7.3, this works with no
> > issues using http, but as soon as I try to use https it will not start.
> >
> >
> >
> > I have tried adding  -Dsolr.http1=true  to the command line but with no
> > joy.
> >
> >
> >
> > Is there some other configuration I can apply to force Solr into using
> > HTTP/1.1 rather than 2 ?
> >
> >
> >
> > Thanks
> >
> >
> >
> > *ALAN HORTON*
> >
> > Senior DevOps Engineer
> >
> > Financial Messaging Marketplaces
> >
> >
> >
> > [image: FINASTRA]
> >
> >
> >
> > Unit E, Nottingham One, 154 Canal Street, Nottingham, NG1 7HG
> >
> > *Email: *alan.hor...@finastra.com
> >
> > *Finastra.com
> > *
> >
> >
> > "FINASTRA" is the trade name of the FINASTRA group of companies. This
> > email and any attachments have been scanned for known viruses using
> > multiple scanners. This email message is intended for the named recipient
> > only. It may be privileged and/or confidential. If you are not the named
> > recipient of this email please notify us immediately and do not copy it or
> > use it for any purpose, nor disclose its contents to any other person. This
> > email does not constitute the commencement of legal relations between you
> > and FINASTRA. Please refer to the executed contract between you and the
> > relevant member of the FINASTRA group for the identity of the contracting
> > party with which you are dealing.
> >
>  
>  
> --
> https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.needhamsoftware.com%2F&data=04%7C01%7CAlan.Horton%40finastra.com%7Cb8e36427094c4056a18708d9be424a0a%7C0b9b90da3fe1457ab340f1b67e1024fb%7C0%7C0%7C637750015038762234%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xCaIB1k2MZwZA5Jl3ZPJEj7AKXhTiKdGxZGTY2i4HOc%3D&reserved=0
>  (work)
> https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.the111shift.com%2F&data=04%7C01%7CAlan.Horton%40finastra.com%7Cb8e36427094c4056a18708d9be424a0a%7C0b9b90da3fe1457ab340f1b67e1024fb%7C0%7C0%7C637750015038762234%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=4De%2F6iZ95hmcTj2dwu7OovM3nYyqoyLkbEbjyegbyEM%3D&reserved=0
>  (play)
>  
> "FINASTRA" is the trade name of the FINASTRA group of companies. This email 
> and any attachments have been scanned for known viruses using multiple 
> scanners. This email message is intended for the named recipient only. It may 
> be privileged and/or confidential. If you are not the named recipient of this 
> email please notify us immediately and do not copy it or use it for any 
> purpose, nor disclose its contents to any other person. This email does not 
> constitute the commencement of legal relations between you and FINASTRA. 
> Please refer to the executed contract between you and the relevant member of 
> the FINASTRA group for the identity of the contracting party with which you 
> are dealing.

Re: [EXT] Re: Starting Solr 8 with Java 8 and SSL

[Gus Heck]

If you were pasting in images, that didn't work. They didn't come through.
Generally it's a bad idea to send images to mailing lists since they are
subject to the whims of how a particular list handles images or
attachments. Furthermore, even if it does come through, you may be getting
people reading on their phones and your image may be way way too large for
their phone to display clearly. Paste the exceptions in the mail directly
or use pastebin service so we all read it and then can likewise copy/paste
the exception into our IDE and instantly navigate to the relevant code.

 Also, you'll save yourself and everyone else a lot of time if you review
this page on how to efficiently ask questions on a mailing list:
http://www.catb.org/~esr/faqs/smart-questions.html especially the sections
titled "Send questions in accessible, standard formats" and "Be precise and
informative about your problem"


On Mon, Dec 13, 2021 at 9:28 AM Horton, Alan
 wrote:

> Initially a series of sessionExpiredExceptions:
> Eventually a timeoutException:
>
> Thanks
>
>
> -Original Message-
> From: Gus Heck <*gus.h...@gmail.com* >
> Sent: 13 December 2021 14:10
> To: *users@solr.apache.org* 
> Subject: [EXT] Re: Starting Solr 8 with Java 8 and SSL
>
> What error message do you get when you try to start solr with https? Can
> you provide more detail as to what you did to "use https"? Hard to know if
> switching to 1.1 would be a good idea here until we understand the problem
> better.
>
> On Mon, Dec 13, 2021 at 6:18 AM Horton, Alan
> <*alan.hor...@finastra.com.invalid* >
> wrote:
>
> > I have updated a Solr install to Solr 8.10 from 7.7.3, this works with no
> > issues using http, but as soon as I try to use https it will not start.
> >
> >
> >
> > I have tried adding  -Dsolr.http1=true  to the command line but with no
> > joy.
> >
> >
> >
> > Is there some other configuration I can apply to force Solr into using
> > HTTP/1.1 rather than 2 ?
> >
> >
> >
> > Thanks
> >
> >
> >
> > *ALAN HORTON*
> >
> > Senior DevOps Engineer
> >
> > Financial Messaging Marketplaces
> >
> >
> >
> > [image: FINASTRA]
> >
> >
> >
> > Unit E, Nottingham One, 154 Canal Street, Nottingham, NG1 7HG
> >
> > *Email: **alan.hor...@finastra.com* <*alan.hor...@finastra.com>
> >
> > *Finastra.com
> > <
> *https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.finastra.com%2F&data=04%7C01%7CAlan.Horton%40finastra.com%7Cb8e36427094c4056a18708d9be424a0a%7C0b9b90da3fe1457ab340f1b67e1024fb%7C0%7C0%7C637750015038762234%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=CFtBgSs83aLznz0HISTjFySXOYVrFg2DkvdM9Zt7Uns%3D&reserved=0*
> 
> >*
> >
> >
> > "FINASTRA" is the trade name of the FINASTRA group of companies. This
> > email and any attachments have been scanned for known viruses using
> > multiple scanners. This email message is intended for the named recipient
> > only. It may be privileged and/or confidential. If you are not the named
> > recipient of this email please notify us immediately and do not copy it
> or
> > use it for any purpose, nor disclose its contents to any other person.
> This
> > email does not constitute the commencement of legal relations between you
> > and FINASTRA. Please refer to the executed contract between you and the
> > relevant member of the FINASTRA group for the identity of the contracting
> > party with which you are dealing.
> >
>
>
> --
>
> *https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.needhamsoftware.com%2F&data=04%7C01%7CAlan.Horton%40finastra.com%7Cb8e36427094c4056a18708d9be424a0a%7C0b9b90da3fe1457ab340f1b67e1024fb%7C0%7C0%7C637750015038762234%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xCaIB1k2MZwZA5Jl3ZPJEj7AKXhTiKdGxZGTY2i4HOc%3D&reserved=0*
> 
> (work)
>
> *https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.the111shift.com%2F&data=04%7C01%7CAlan.Horton%40finastra.com%7Cb8e36427094c4056a18708d9be424a0a%7C0b9b90da3fe1457ab340f1b67e1024fb%7C0%7C0%7C637750015038762234%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=4De%2F6iZ95h

Re: Solr Cloud Node re-join issue

[Walter Underwood]

The heap size limits the Java heap. Java uses memory that is not in the heap — 
bytecodes, compiled code, stack, threadlocal, mapped memory, etc.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Dec 13, 2021, at 7:53 AM, Scott  wrote:
> 
> So you're saying that Solr can use additional memory on top of what Xmx 
> limits ? It appears the resident size keeps on increasing, swap was used at 
> some point but it's not actively paging now.
> 
> In FreeBSD you can use procstat or vmstat for more info
> 
> root@solrcloud4:/usr/home/scott # procstat -r 57116
>  PID COMM RESOURCE  VALUE
> 57116 java user time07:59:00.567533
> 57116 java system time  00:31:36.921875
> 57116 java maximum RSS 31175620 KB
> 57116 java integral shared memory 139301820 KB
> 57116 java integral unshared data  46433940 KB
> 57116 java integral unshared stack495295360 KB
> 57116 java page reclaims   11361433
> 57116 java page faults  2919735
> 57116 java swaps  0
> 57116 java block reads  2918179
> 57116 java block writes 1724654
> 57116 java messages sent   23464275
> 57116 java messages received   13276352
> 57116 java signals received   28619
> 57116 java voluntary context switches  66989710
> 57116 java involuntary context switches49835294
> 
> So with a SOLR_HEAP value of 16g , I'm now at a total size of 30g which has 
> already used some swap that it hasn't released yet. This will keep on going 
> until swap usage is 100% and the box will crash.
> 
> I guess my questions are:
> - Why does Solr use more than 16g ?
> - Why isn't swapped memory released ?
> 
> Thanks!
> Scott
> 
> -Original Message-
> From: Shawn Heisey  
> Sent: Monday, December 13, 2021 12:41 AM
> To: users@solr.apache.org
> Subject: Re: Solr Cloud Node re-join issue
> 
> On 12/12/2021 4:40 PM, Scott wrote:
>> However, top still shows Solr using more than 16g . It started at 17g 
>> and has been steadily growing, now it's at 23g and soon it will go 
>> into swap
>> 
>> PID  USERNAMETHR PRI NICE   SIZE  RES  SWAP STATEC   TIME
>> WCPU COMMAND
>> 57116 solr  165  520   235G23G0   uwait  
>>   1  36:20   9.60% java
> 
> I have been doing some experiments with a FreeBSD VM running in vmware 
> player.  I have very little experience with that OS.
> 
> I have openjdk11 and apache-solr-8.10.0 installed using pkg.
> 
> It looks like top in FreeBSD has no equivalent to the SHR column seen on 
> Linux.  Being able to see how much shared memory is being used is critical to 
> seeing a complete picture of memory usage.  I suspect that if we could see 
> it, the shared memory would be approximately 7GB when the RES column says 
> 23GB.  This is something I have seen on Linux, and I have deduced that the 
> actual memory used by the process will be the RES size minus the SHR size.  
> Sometimes the shared memory will get quite large.  I have no idea why this 
> happens, but it does.
> 
> There is a Java tool called "jstat" that can give a very accurate picture of 
> Java program memory usage.  But when the -XX:+PerfDisableSharedMem option is 
> given to Java, that tool doesn't work.  That option is added with the default 
> GC tuning options, because it eliminates a severe performance issue that is 
> sometimes seen with Java software.
> 
> If you add the following to solr.in.sh then jstat will work:
> 
> GC_TUNE="-XX:+UseG1GC \
>   -XX:+ParallelRefProcEnabled \
>   -XX:MaxGCPauseMillis=250 \
>   -XX:+UseLargePages \
>   -XX:+AlwaysPreTouch \
>   -XX:+ExplicitGCInvokesConcurrent"
> 
> After adding it, restart Solr and use the following command with the PID of 
> the Solr process in place of PID, at a time when the RES column for Solr goes 
> well beyond 16GB:
> 
> sudo jstat -gc -t PID 5000 20 > /tmp/jstat.out
> 
> That command will take a little less than two minutes to complete.  Then you 
> can share the /tmp/jstat.out file using a file sharing website. 
> Don't try to paste it into email ... the lines are VERY long.
> 
> If you add up the columns named S0C, S1C, EC, OC, MC, and CCSC for a given 
> line, that will be pretty close to the process's total memory usage, in KB.
> 
> If you want to know what all those columns mean, here's Oracle's 
> documentation for jstat:
> 
> https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html
> 
> I've just gotten a look at the output from vmstat ... looks like that tool is 
> useless for what I was trying 

RE: Solr Cloud Node re-join issue

[Scott]

I get this, so then is my experience similar to everyone else's ? Does swap 
usage just increase until you run out of it ? I mean, I've always taken this to 
be true and that's why I restart Solr once per month ...

Is there a way to not have to do this ? That's how this thread all got started.

Thanks!

-Original Message-
From: Walter Underwood  
Sent: Monday, December 13, 2021 12:24 PM
To: users@solr.apache.org
Subject: Re: Solr Cloud Node re-join issue

The heap size limits the Java heap. Java uses memory that is not in the heap — 
bytecodes, compiled code, stack, threadlocal, mapped memory, etc.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Dec 13, 2021, at 7:53 AM, Scott  wrote:
> 
> So you're saying that Solr can use additional memory on top of what Xmx 
> limits ? It appears the resident size keeps on increasing, swap was used at 
> some point but it's not actively paging now.
> 
> In FreeBSD you can use procstat or vmstat for more info
> 
> root@solrcloud4:/usr/home/scott # procstat -r 57116
>  PID COMM RESOURCE  VALUE
> 57116 java user time07:59:00.567533
> 57116 java system time  00:31:36.921875
> 57116 java maximum RSS 31175620 KB
> 57116 java integral shared memory 139301820 KB
> 57116 java integral unshared data  46433940 KB
> 57116 java integral unshared stack495295360 KB
> 57116 java page reclaims   11361433
> 57116 java page faults  2919735
> 57116 java swaps  0
> 57116 java block reads  2918179
> 57116 java block writes 1724654
> 57116 java messages sent   23464275
> 57116 java messages received   13276352
> 57116 java signals received   28619
> 57116 java voluntary context switches  66989710
> 57116 java involuntary context switches49835294
> 
> So with a SOLR_HEAP value of 16g , I'm now at a total size of 30g which has 
> already used some swap that it hasn't released yet. This will keep on going 
> until swap usage is 100% and the box will crash.
> 
> I guess my questions are:
> - Why does Solr use more than 16g ?
> - Why isn't swapped memory released ?
> 
> Thanks!
> Scott
> 
> -Original Message-
> From: Shawn Heisey 
> Sent: Monday, December 13, 2021 12:41 AM
> To: users@solr.apache.org
> Subject: Re: Solr Cloud Node re-join issue
> 
> On 12/12/2021 4:40 PM, Scott wrote:
>> However, top still shows Solr using more than 16g . It started at 17g 
>> and has been steadily growing, now it's at 23g and soon it will go 
>> into swap
>> 
>> PID  USERNAMETHR PRI NICE   SIZE  RES  SWAP STATEC   TIME
>> WCPU COMMAND
>> 57116 solr  165  520   235G23G0   uwait  
>>   1  36:20   9.60% java
> 
> I have been doing some experiments with a FreeBSD VM running in vmware 
> player.  I have very little experience with that OS.
> 
> I have openjdk11 and apache-solr-8.10.0 installed using pkg.
> 
> It looks like top in FreeBSD has no equivalent to the SHR column seen on 
> Linux.  Being able to see how much shared memory is being used is critical to 
> seeing a complete picture of memory usage.  I suspect that if we could see 
> it, the shared memory would be approximately 7GB when the RES column says 
> 23GB.  This is something I have seen on Linux, and I have deduced that the 
> actual memory used by the process will be the RES size minus the SHR size.  
> Sometimes the shared memory will get quite large.  I have no idea why this 
> happens, but it does.
> 
> There is a Java tool called "jstat" that can give a very accurate picture of 
> Java program memory usage.  But when the -XX:+PerfDisableSharedMem option is 
> given to Java, that tool doesn't work.  That option is added with the default 
> GC tuning options, because it eliminates a severe performance issue that is 
> sometimes seen with Java software.
> 
> If you add the following to solr.in.sh then jstat will work:
> 
> GC_TUNE="-XX:+UseG1GC \
>   -XX:+ParallelRefProcEnabled \
>   -XX:MaxGCPauseMillis=250 \
>   -XX:+UseLargePages \
>   -XX:+AlwaysPreTouch \
>   -XX:+ExplicitGCInvokesConcurrent"
> 
> After adding it, restart Solr and use the following command with the PID of 
> the Solr process in place of PID, at a time when the RES column for Solr goes 
> well beyond 16GB:
> 
> sudo jstat -gc -t PID 5000 20 > /tmp/jstat.out
> 
> That command will take a little less than two minutes to complete.  Then you 
> can share the /tmp/jstat.out file using a file sharing website. 
> Don't try to paste it into email ... the l

Re: Solr Cloud Node re-join issue

[Walter Underwood]

A monthly restart should not be necessary. We did a quarterly restart for one 
Solr 6.x cluster, but haven’t done that for our 8.7 clusters running on Java 
11. 

I was going to check how long ours have been up, but every Solr server was 
restarted 3 days ago to fix the log4j vulnerability.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Dec 13, 2021, at 9:36 AM, Scott  wrote:
> 
> I get this, so then is my experience similar to everyone else's ? Does swap 
> usage just increase until you run out of it ? I mean, I've always taken this 
> to be true and that's why I restart Solr once per month ...
> 
> Is there a way to not have to do this ? That's how this thread all got 
> started.
> 
> Thanks!
> 
> -Original Message-
> From: Walter Underwood  
> Sent: Monday, December 13, 2021 12:24 PM
> To: users@solr.apache.org
> Subject: Re: Solr Cloud Node re-join issue
> 
> The heap size limits the Java heap. Java uses memory that is not in the heap 
> — bytecodes, compiled code, stack, threadlocal, mapped memory, etc.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
>> On Dec 13, 2021, at 7:53 AM, Scott  wrote:
>> 
>> So you're saying that Solr can use additional memory on top of what Xmx 
>> limits ? It appears the resident size keeps on increasing, swap was used at 
>> some point but it's not actively paging now.
>> 
>> In FreeBSD you can use procstat or vmstat for more info
>> 
>> root@solrcloud4:/usr/home/scott # procstat -r 57116
>> PID COMM RESOURCE  VALUE
>> 57116 java user time07:59:00.567533
>> 57116 java system time  00:31:36.921875
>> 57116 java maximum RSS 31175620 KB
>> 57116 java integral shared memory 139301820 KB
>> 57116 java integral unshared data  46433940 KB
>> 57116 java integral unshared stack495295360 KB
>> 57116 java page reclaims   11361433
>> 57116 java page faults  2919735
>> 57116 java swaps  0
>> 57116 java block reads  2918179
>> 57116 java block writes 1724654
>> 57116 java messages sent   23464275
>> 57116 java messages received   13276352
>> 57116 java signals received   28619
>> 57116 java voluntary context switches  66989710
>> 57116 java involuntary context switches49835294
>> 
>> So with a SOLR_HEAP value of 16g , I'm now at a total size of 30g which has 
>> already used some swap that it hasn't released yet. This will keep on going 
>> until swap usage is 100% and the box will crash.
>> 
>> I guess my questions are:
>> - Why does Solr use more than 16g ?
>> - Why isn't swapped memory released ?
>> 
>> Thanks!
>> Scott
>> 
>> -Original Message-
>> From: Shawn Heisey 
>> Sent: Monday, December 13, 2021 12:41 AM
>> To: users@solr.apache.org
>> Subject: Re: Solr Cloud Node re-join issue
>> 
>> On 12/12/2021 4:40 PM, Scott wrote:
>>> However, top still shows Solr using more than 16g . It started at 17g 
>>> and has been steadily growing, now it's at 23g and soon it will go 
>>> into swap
>>> 
>>> PID  USERNAMETHR PRI NICE   SIZE  RES  SWAP STATEC   TIME   
>>>  WCPU COMMAND
>>> 57116 solr  165  520   235G23G0   uwait 
>>>1  36:20   9.60% java
>> 
>> I have been doing some experiments with a FreeBSD VM running in vmware 
>> player.  I have very little experience with that OS.
>> 
>> I have openjdk11 and apache-solr-8.10.0 installed using pkg.
>> 
>> It looks like top in FreeBSD has no equivalent to the SHR column seen on 
>> Linux.  Being able to see how much shared memory is being used is critical 
>> to seeing a complete picture of memory usage.  I suspect that if we could 
>> see it, the shared memory would be approximately 7GB when the RES column 
>> says 23GB.  This is something I have seen on Linux, and I have deduced that 
>> the actual memory used by the process will be the RES size minus the SHR 
>> size.  Sometimes the shared memory will get quite large.  I have no idea why 
>> this happens, but it does.
>> 
>> There is a Java tool called "jstat" that can give a very accurate picture of 
>> Java program memory usage.  But when the -XX:+PerfDisableSharedMem option is 
>> given to Java, that tool doesn't work.  That option is added with the 
>> default GC tuning options, because it eliminates a severe performance issue 
>> that is sometimes seen with Java software.
>> 
>> If you add the following to solr.in.sh then jstat will work:
>> 
>> GC_TUNE="-XX:+UseG1GC \
>>  -XX:+ParallelRefProcEnabl

When will solr 8.11.1 become available?

[Tang, Rebecca]

Thanks,
Rebecca

Re: Solr Cloud Node re-join issue

[Shawn Heisey]

On 12/13/21 8:53 AM, Scott wrote:

I guess my questions are:
- Why does Solr use more than 16g ?
- Why isn't swapped memory released ?


Solr is a Java program, and it is Java that manages the memory. If the 
process is going significantly beyond the 16GB max heap that has been 
configured, and swap is in use, then it is most likely Java that is 
broken.  Java simply shouldn't use much more memory than it has been 
told it can use.  Some overhead beyond the configured max heap will 
always be required, but typically the overhead should be pretty small.  
Maybe a few hundred MB when the heap is 16GB.


Swap is managed by the OS, not Java or Solr.  A significant bug in the 
OS seems unlikely, but that sort of thing HAS happened before.  Do you 
have all the packages on the server, including the kernel and Java, 
fully up to date?


All this assumes that there are no other processes on the machine that 
are using significant amounts of memory.


If you can't get an updated version of Java 11, what I would try next is 
installing openjdk8 and setting JAVA_HOME in solr.in.sh so it points at 
that version of Java.  Java 8 is the minimum requirement for Solr 8.x.  
It has been out a lot longer than Java 11, which in theory would mean 
it's less likely to be affected by major bugs.


Something seems to be very broken somewhere, either in Java or the OS.  
I would suspect Java before the OS.  It just shouldn't behave like 
this.  On a well-tuned system that is not suffering from significant 
bugs, it should never be necessary to restart Solr. Solr has not had 
valid reports of a memory leak bug in a long time... but even if Solr 
did have a memory leak, that would not cause Java to use so much extra 
memory.


Thanks,
Shawn

Re: When will solr 8.11.1 become available?

[Jan Høydahl]

You can follow the release on the dev@ mailing list, see 
https://lists.apache.org/list.html?d...@solr.apache.org

There will be a first release candidate soon-ish, and depending on voting 
outcome there may be more RCs during the week and then finally an official 
release.

Jan

> 13. des. 2021 kl. 20:53 skrev Tang, Rebecca :
> 
> Thanks,
> Rebecca

Re: Log4j vulnerability- Solr4 - urgent pls

[Raveendra Yerraguntla]

For Solr 4, you don’t need this

Please see other replies

Sent from my iPhone

> On Dec 12, 2021, at 12:50 AM, Reej Nayagam  wrote:
> 
> Thanks for the reply.
> 
> *REgards,*
> *Reej*
> 
> 
>> On Sun, Dec 12, 2021 at 12:28 PM Rahul Goswami 
>> wrote:
>> 
>> In case of solr4 which uses log4j-1.2.17.jar, the
>> "log4j2.formatMsgNoLookups=true" system property is neither required nor
>> applicable. In fact, the property was only introduced in log4j-2.10 (refer
>> to the JIRA below). So not just Solr, but any Java application using 2<=
>> log4j <2.10 will not be helped by this system property.
>> 
>> https://issues.apache.org/jira/browse/LOG4J2-2109
>> 
>> On Sat, Dec 11, 2021 at 11:04 PM Raveendra Yerraguntla
>>  wrote:
>> 
>>> 
>>>   - -Dlog4j2.formatMsgNoLookups=true
>>> 
>>> 
>>> restart jvm with the above param and should work.
>>> 
>>> 
>>> 
>>> 
>>> 
>>>On Saturday, December 11, 2021, 09:51:54 PM EST, Reej Nayagam <
>>> reej...@gmail.com> wrote:
>>> 
>>> Hi All,
>>> 
>>> In production we are using solr4 which uses log4j-1.2.17.jar.
>>> 
>>> Can someone say the mitigation option for solr4
>>> 
>>> Thanks
>>> Reej
>>> --
>>> *Thanks,*
>>> *Reej*
>>> 
>> 

Re: When will solr 8.11.1 become available?

[Shawn Heisey]

On 12/13/21 12:53 PM, Tang, Rebecca wrote:

Thanks,
Rebecca



It is impossible to give you an accurate prediction for the release date 
of Solr 8.11.1.


Yesterday, the developer who designated themselves the release manager 
for 8.11.1 said on the Solr dev list that they would be creating the 
first release candidate soon.  I would expect it before the end of the 
week, but cannot guarantee that.


Once a release candidate is available, it will then go to a vote about 
whether or not that candidate should be released.  Project rules dictate 
that the vote will run for 72 hours.  Assuming the vote passes, the 
process of making the official release available on the Apache mirror 
system will begin once the vote concludes.  At that point it can be 
downloaded from the official archive.  It usually takes a day or two 
before enough mirrors have the release that the project feels 
comfortable announcing it.


If the vote doesn't pass, then whatever problem was found must be fixed 
and a new release candidate created, and then a brand new 72-hour vote 
will commence.


I believe that votes for 8.x releases will happen on the Lucene dev 
list, because for 8.x Solr is still part of the lucene-solr codebase.  
When Solr 9.0 reaches the point where it can be released, it will be 
released from a codebase that only includes Solr, and so the vote should 
move to the Solr dev list.


Thanks,
Shawn

Re: When will solr 8.11.1 become available?

[Andy Lester]

> It is impossible to give you an accurate prediction for the release date of 
> Solr 8.11.1.

It sounds like it’s safe to say that the release will be “on the order of at 
least a week from now” would be safe, right? 

That might be all the accuracy that someone needs.

Re: When will solr 8.11.1 become available?

[Shawn Heisey]

On 12/13/2021 4:12 PM, Andy Lester wrote:

It is impossible to give you an accurate prediction for the release date of 
Solr 8.11.1.


It sounds like it’s safe to say that the release will be “on the order of at 
least a week from now” would be safe, right?

That might be all the accuracy that someone needs.


If everything goes well and no problems are discovered with the first 
release candidate, that would likely be about when the release is announced.


Thanks,
Shawn