Re: Solr and CVE-2021-44228

[Bram Van Dam]

In case anyone wants to patch 7.7.3 from source, here's a patch and 
quick build instructions:


Apply the attached patch -- hopefully the mailing list won't nerf the 
attachment.


git am < /path/to/CVE-2021-4422.txt
ant clean compile jar -Dversion=7.7.3
cd solr
ant package -Dversion=7.7.3


 - BramFrom: Bram 
Date: Fri, 10 Dec 2021 13:44:29 +0100
Subject: [PATCH] CVE-2021-44228 Critical security issue in Log4J

---
 lucene/ivy-versions.properties | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/lucene/ivy-versions.properties b/lucene/ivy-versions.properties
index 1c6be1ebf13..a8eca1a18ae 100644
--- a/lucene/ivy-versions.properties
+++ b/lucene/ivy-versions.properties
@@ -178,7 +178,7 @@ org.apache.james.apache.mime4j.version = 0.8.2
 /org.apache.james/apache-mime4j-core = 
${org.apache.james.apache.mime4j.version}
 /org.apache.james/apache-mime4j-dom = ${org.apache.james.apache.mime4j.version}
 
-org.apache.logging.log4j.version = 2.11.0
+org.apache.logging.log4j.version = 2.15.0
 /org.apache.logging.log4j/log4j-1.2-api = ${org.apache.logging.log4j.version}
 /org.apache.logging.log4j/log4j-api = ${org.apache.logging.log4j.version}
 /org.apache.logging.log4j/log4j-core = ${org.apache.logging.log4j.version}
-- 
2.31.1

Update Log4J library version for Solr 6.6.3

[Woei Jong Yoon]

Hi All,

Currently Solr version 6.6.3 are using log4j library with version 1.2.17.

If we plan to update the log4j library version to 2.15 due the log4j library is 
end of support.

May we check that Solr 6.6.3 able to support it?

Additional advice will be appreciated.

Thank you.

Regards,
Yoon Woei Jong


www.xtremax.coml   114 Lavender Street #08-93 CT 
Hub 2 Singapore 338729


CONFIDENTIALITY NOTICE:

The contents of this email message and any attachments are intended solely for 
the addressee(s) and may contain confidential and/or privileged information and 
may be legally protected from disclosure. If you are not the intended recipient 
of this message or their agent, or if this message has been addressed to you in 
error, please immediately alert the sender by reply e-mail and then delete this 
message and any attachments. If you are not the intended recipient, you are 
hereby notified that any use, dissemination, copying, or storage of this 
message or its attachments is strictly prohibited.

Re: Update Log4J library version for Solr 6.6.3

[Yuval Paz]

Log4j2 is not a simple upgrade of log4j(1), in fact, it is a completely new
library.

Although I'm not 100% sure about how Solr uses log4j, in most cases it is
not a simple plug and play.

If you are concerned about the 0-day vulnerability, see the link below.

>From my understanding, the vulnerability only affected log4j2

https://github.com/apache/logging-log4j2/pull/608#issuecomment-990494126

On Sat, Dec 11, 2021, 1:49 PM Woei Jong Yoon  wrote:

> Hi All,
>
> Currently Solr version 6.6.3 are using log4j library with version 1.2.17.
>
> If we plan to update the log4j library version to 2.15 due the log4j
> library is end of support.
>
> May we check that Solr 6.6.3 able to support it?
>
> Additional advice will be appreciated.
>
> Thank you.
>
> Regards,
> Yoon Woei Jong
>
>
> www.xtremax.coml   114 Lavender Street
> #08-93 CT Hub 2 Singapore 338729
>
>
> CONFIDENTIALITY NOTICE:
>
> The contents of this email message and any attachments are intended solely
> for the addressee(s) and may contain confidential and/or privileged
> information and may be legally protected from disclosure. If you are not
> the intended recipient of this message or their agent, or if this message
> has been addressed to you in error, please immediately alert the sender by
> reply e-mail and then delete this message and any attachments. If you are
> not the intended recipient, you are hereby notified that any use,
> dissemination, copying, or storage of this message or its attachments is
> strictly prohibited.
>

RE: Solr Cloud Node re-join issue

[Scott]

Thanks Shawn. Most people I talked to sort of acknowledged that restarting SOLR 
every month or so is a given, but your comments are encouraging.

These nodes have 32Gb of ram:

real memory  = 34359738368 (32768 MB)
avail memory = 33370628096 (31824 MB)

and here's what I have in my solr config

SOLR_JAVA_MEM="-Xms14512m -Xmx16512m"

I thought I'd keep it at half of available RAM but it still goes into swap...

Thank you for your help

-Original Message-
From: Shawn Heisey  
Sent: Saturday, December 11, 2021 1:25 AM
To: users@solr.apache.org
Subject: Re: Solr Cloud Node re-join issue

On 12/10/2021 12:38 PM, Scott wrote:
> Having a bit of  weird issue.
> 
> We run a 4 node Solr Cloud , version 8.6.2 and for the most part it's 
> been going quite well for more than 2 years now. We have to restart 
> them occasionally to free up ram but I guess that's normal.

If you have to restart because it's using too much memory, then something is 
not configured right.  If the java heap is sized appropriately, and the machine 
is not being used to handle software other than Solr, it is pretty much 
impossible for a java program like Solr to take too much memory.

> Last night one of the nodes went into swap, used up all memory and crashed.
> Somehow the way it crashed, it also removed all local cores/data. The 
> cluster kept on chugging along which was fine, but now I can't get the 
> crashed node to resync with the others.

Assuming again that Solr is the only significant memory-using process on the 
system, and the heap is sized appropriately, then that system should NEVER use 
significant amounts of swap.

I'm betting that you have configured Solr with a max heap size that's too large 
for the system it's running on.  Because Java uses a garbage collection memory 
model, almost any Java program will eventually use the entire max heap size it 
has been given, even if it does not actually need that much memory.  This is 
expected.

The most likely reason for a SolrCloud node to delete all cores is that it 
connects to a zookeeper ensemble that does not contain SolrCloud cluster config 
data, or contains a cluster config that's not the correct one.  See this issue:

https://issues.apache.org/jira/browse/SOLR-13396

Thanks,
Shawn



This is a private message

Re: Solr Cloud Node re-join issue

[Shawn Heisey]

On 12/11/2021 8:39 AM, Scott wrote:

Thanks Shawn. Most people I talked to sort of acknowledged that restarting SOLR 
every month or so is a given, but your comments are encouraging.

These nodes have 32Gb of ram:

real memory  = 34359738368 (32768 MB)
avail memory = 33370628096 (31824 MB)

and here's what I have in my solr config

SOLR_JAVA_MEM="-Xms14512m -Xmx16512m"


I have noticed with Linux servers under my care that they do utilize 
more swap than I would have ever expected.  But most of them seem to be 
healthy despite that.


If your machines are running Linux, then the following will apply.  For 
other UNIX-like operating systems, I have no idea whether you can use 
this info:


---
This shell script will show you what programs are utilizing swap.  It 
must be run as root:


https://apaste.info/Gnwx

It's not actually important whether there is swap in use.  What's really 
important is whether or not the system is ACTIVELY swapping.  You can 
see that with the following command, which I recommend running with a 
terminal width well beyond 80:


vmstat 2 -w

If the si and so columns frequently show numbers other than 0, then 
there is active swapping going on.  If it is almost always 0 or a very 
small number, then there is probably no need to worry about swap usage.

---

With the "top" utility on most UNIX-like operating systems, there is a 
way we can see the overall health of memory usage on a system.  If my 
email program clobbers this link, I apologize:


https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue


I thought I'd keep it at half of available RAM but it still goes into swap...


I really want to smack the person who came up with the rule of thumb 
where a Java heap gets assigned to be some fraction of the total memory 
size, usually between 25 and 50 percent.  In practice, that advice 
usually means that the heap will be too small or significantly larger 
than it needs to be.


What you should do for the heap size is figure out how much heap memory 
the program actually needs to do its work, then assign the max heap to a 
number above that which gives the program some headroom to operate in. 
I do not have any generic guidelines for how much headroom is needed ... 
it will depend on the exact situation you find yourself in.


We also recommend setting the min heap and max heap to the same number. 
 For your system that would mean this (instead of SOLR_JAVA_MEM):


SOLR_HEAP="16g"

Thanks,
Shawn

Re: Update Log4J library version for Solr 6.6.3

[Walter Underwood]

log4j 1.x does not have the vulnerability, so you do not need to patch 6.6.3.

If you want a current, non-vulnerable log4j library, you will need to upgrade 
to Solr 8.11.1.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Dec 11, 2021, at 3:34 AM, Woei Jong Yoon  wrote:
> 
> Hi All,
> 
> Currently Solr version 6.6.3 are using log4j library with version 1.2.17.
> 
> If we plan to update the log4j library version to 2.15 due the log4j library 
> is end of support.
> 
> May we check that Solr 6.6.3 able to support it?
> 
> Additional advice will be appreciated.
> 
> Thank you.
> 
> Regards,
> Yoon Woei Jong
> 
> 
> www.xtremax.coml   114 Lavender Street #08-93 CT 
> Hub 2 Singapore 338729
> 
> 
> CONFIDENTIALITY NOTICE:
> 
> The contents of this email message and any attachments are intended solely 
> for the addressee(s) and may contain confidential and/or privileged 
> information and may be legally protected from disclosure. If you are not the 
> intended recipient of this message or their agent, or if this message has 
> been addressed to you in error, please immediately alert the sender by reply 
> e-mail and then delete this message and any attachments. If you are not the 
> intended recipient, you are hereby notified that any use, dissemination, 
> copying, or storage of this message or its attachments is strictly prohibited.

log4j zero day exploit

[Scott Derrick]

Trying to mitigate the zero day log4j exploit without upgrading my solr 
instance


per 
https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228


I  made the following edits  :
    (Linux/MacOS) Edit your |solr.in.sh| file to include: 
|SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true"


I restarted solr but would like to verify my instance is running with 
the log4j2 setting.


I can't figure out how to see what SOLR_OPTS it started with?

thanks,

Scott
|

log4j zero day exploit

[Scott Derrick]

Trying to mitigate the zero day log4j exploit without upgrading my solr 
instance


per 
https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228


I  made the following edits  :
    (Linux/MacOS) Edit your |solr.in.sh| file to include: 
|SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true"


I restarted solr but would like to verify my instance is running with 
the log4j2 setting.


I can't figure out how to see what SOLR_OPTS it started with?

thanks,

Scott
|

log4j2 exploit

[Scott Derrick]

figured out how to echo the solr_opts in startup

Scott

Re: log4j zero day exploit

[Shawn Heisey]

On 12/11/21 2:05 PM, Scott Derrick wrote:
Trying to mitigate the zero day log4j exploit without upgrading my 
solr instance


per 
https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228


I  made the following edits  :
    (Linux/MacOS) Edit your |solr.in.sh| file to include: 
|SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true"


On my 8.11.0 server, I replaced all the log4j jars in server/lib/ext 
(which were the 2.14.1 version) with the 2.15.0 versions.  Solr is still 
working after restarting.  My Solr install isn't reachable by anyone 
outside of the machine itself, so I don't worry too much about 
vulnerabilities.  If somebody breaches the server, they will already be 
able to see and affect far more than what's in my Solr index.


Updating jars in this way is something that does not always work. 
Sometimes a dependency update will require changes to Solr's source 
code.  This is one instance where no code changes were required.


I restarted solr but would like to verify my instance is running with 
the log4j2 setting.


I can't figure out how to see what SOLR_OPTS it started with? 



Open the admin UI and look at the dashboard.  It will give you all the 
commandline JVM args that Solr was started with.  If you see the "-D" 
option that you added, you're good.


You might also be able to see with "ps auxww | grep solr" which I know 
works on Linux.  Other operating systems might need different args for ps.


Thanks,
Shawn

Re: log4j zero day exploit

[Walter Underwood]

The startup options are shown on the home admin page, in alphabetical order. 
Very handy.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Dec 11, 2021, at 1:09 PM, Scott Derrick  wrote:
> 
> Trying to mitigate the zero day log4j exploit without upgrading my solr 
> instance
> 
> per 
> https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228
> 
> I  made the following edits  :
> (Linux/MacOS) Edit your |solr.in.sh| file to include: 
> |SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true"
> 
> I restarted solr but would like to verify my instance is running with the 
> log4j2 setting.
> 
> I can't figure out how to see what SOLR_OPTS it started with?
> 
> thanks,
> 
> Scott
> |

Re: log4j zero day exploit

[Tim Casey]

The vulnerability is quite nasty.  If there is a user string logged in a
log4j line, then you are vulnerable.
I would suspect everyone would need to at least worry about it or risk
becoming a bitcoin harvester.

tim


On Sat, Dec 11, 2021 at 2:19 PM Shawn Heisey  wrote:

> On 12/11/21 2:05 PM, Scott Derrick wrote:
> > Trying to mitigate the zero day log4j exploit without upgrading my
> > solr instance
> >
> > per
> >
> https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228
> >
> > I  made the following edits  :
> > (Linux/MacOS) Edit your |solr.in.sh| file to include:
> > |SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true"
>
> On my 8.11.0 server, I replaced all the log4j jars in server/lib/ext
> (which were the 2.14.1 version) with the 2.15.0 versions.  Solr is still
> working after restarting.  My Solr install isn't reachable by anyone
> outside of the machine itself, so I don't worry too much about
> vulnerabilities.  If somebody breaches the server, they will already be
> able to see and affect far more than what's in my Solr index.
>
> Updating jars in this way is something that does not always work.
> Sometimes a dependency update will require changes to Solr's source
> code.  This is one instance where no code changes were required.
>
> > I restarted solr but would like to verify my instance is running with
> > the log4j2 setting.
> >
> > I can't figure out how to see what SOLR_OPTS it started with?
>
>
> Open the admin UI and look at the dashboard.  It will give you all the
> commandline JVM args that Solr was started with.  If you see the "-D"
> option that you added, you're good.
>
> You might also be able to see with "ps auxww | grep solr" which I know
> works on Linux.  Other operating systems might need different args for ps.
>
> Thanks,
> Shawn
>
>
>

Log4j vulnerability- Solr4 - urgent pls

[Reej Nayagam]

Hi All,

In production we are using solr4 which uses log4j-1.2.17.jar.

Can someone say the mitigation option for solr4

Thanks
Reej
-- 
*Thanks,*
*Reej*

Re: Log4j vulnerability- Solr4 - urgent pls

[Rahul Goswami]

As pointed out by the author of log4j 1.x, the library is not susceptible
to this attack the way log4j2 is.
https://github.com/apache/logging-log4j2/pull/608#issuecomment-991380319

So you should be good.

Rahul

On Sat, Dec 11, 2021 at 9:51 PM Reej Nayagam  wrote:

> Hi All,
>
> In production we are using solr4 which uses log4j-1.2.17.jar.
>
> Can someone say the mitigation option for solr4
>
> Thanks
> Reej
> --
> *Thanks,*
> *Reej*
>

Re: Log4j vulnerability- Solr4 - urgent pls

[Raveendra Yerraguntla]

   
   - -Dlog4j2.formatMsgNoLookups=true   


restart jvm with the above param and should work.



 

On Saturday, December 11, 2021, 09:51:54 PM EST, Reej Nayagam 
 wrote:  
 
 Hi All,

In production we are using solr4 which uses log4j-1.2.17.jar.

Can someone say the mitigation option for solr4

Thanks
Reej
-- 
*Thanks,*
*Reej*
  

Re: Log4j vulnerability- Solr4 - urgent pls

[Rahul Goswami]

In case of solr4 which uses log4j-1.2.17.jar, the
"log4j2.formatMsgNoLookups=true" system property is neither required nor
applicable. In fact, the property was only introduced in log4j-2.10 (refer
to the JIRA below). So not just Solr, but any Java application using 2<=
log4j <2.10 will not be helped by this system property.

https://issues.apache.org/jira/browse/LOG4J2-2109

On Sat, Dec 11, 2021 at 11:04 PM Raveendra Yerraguntla
 wrote:

>
>- -Dlog4j2.formatMsgNoLookups=true
>
>
> restart jvm with the above param and should work.
>
>
>
>
>
> On Saturday, December 11, 2021, 09:51:54 PM EST, Reej Nayagam <
> reej...@gmail.com> wrote:
>
>  Hi All,
>
> In production we are using solr4 which uses log4j-1.2.17.jar.
>
> Can someone say the mitigation option for solr4
>
> Thanks
> Reej
> --
> *Thanks,*
> *Reej*
>

Re: Log4j vulnerability- Solr4 - urgent pls

[Reej Nayagam]

Thank you for your reply.

It mentions Dlog4j2 but with solr4 it is log4j1.2.17
Can we use this command

   - -*Dlog4j2*.formatMsgNoLookups=true


On Sun, 12 Dec 2021 at 12:03 PM, Raveendra Yerraguntla
 wrote:

>
>- -Dlog4j2.formatMsgNoLookups=true
>
>
> restart jvm with the above param and should work.
>
>
>
>
>
> On Saturday, December 11, 2021, 09:51:54 PM EST, Reej Nayagam <
> reej...@gmail.com> wrote:
>
>  Hi All,
>
> In production we are using solr4 which uses log4j-1.2.17.jar.
>
> Can someone say the mitigation option for solr4
>
> Thanks
> Reej
> --
> *Thanks,*
> *Reej*
>

-- 
*Thanks,*
*Reej*

Re: Log4j vulnerability- Solr4 - urgent pls

[Walter Underwood]

Solr 4 does NOT have the vulnerability. You do not have to do anything.

From the Solr Security page:

2021-12-10, Apache Solr affected by Apache Log4J CVE-2021-44228 

Severity: Critical

Versions Affected: 7.4.0 to 7.7.3, 8.0.0 to 8.11.0

https://solr.apache.org/security.html 

Solr 4 is before Solr 7.4, so it is not affected by this problem.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Dec 11, 2021, at 8:28 PM, Reej Nayagam  wrote:
> 
> Thank you for your reply.
> 
> It mentions Dlog4j2 but with solr4 it is log4j1.2.17
> Can we use this command
> 
>   - -*Dlog4j2*.formatMsgNoLookups=true
> 
> 
> On Sun, 12 Dec 2021 at 12:03 PM, Raveendra Yerraguntla
>  wrote:
> 
>> 
>>   - -Dlog4j2.formatMsgNoLookups=true
>> 
>> 
>> restart jvm with the above param and should work.
>> 
>> 
>> 
>> 
>> 
>>On Saturday, December 11, 2021, 09:51:54 PM EST, Reej Nayagam <
>> reej...@gmail.com> wrote:
>> 
>> Hi All,
>> 
>> In production we are using solr4 which uses log4j-1.2.17.jar.
>> 
>> Can someone say the mitigation option for solr4
>> 
>> Thanks
>> Reej
>> --
>> *Thanks,*
>> *Reej*
>> 
> 
> -- 
> *Thanks,*
> *Reej*

Re: Log4j vulnerability- Solr4 - urgent pls

[Reej Nayagam]

Thank you for the reply.

*Thanks,*
*Reej*


On Sun, Dec 12, 2021 at 12:38 PM Walter Underwood 
wrote:

> Solr 4 does NOT have the vulnerability. You do not have to do anything.
>
> From the Solr Security page:
>
> 2021-12-10, Apache Solr affected by Apache Log4J CVE-2021-44228
>
> Severity: Critical
>
> Versions Affected: 7.4.0 to 7.7.3, 8.0.0 to 8.11.0
>
> https://solr.apache.org/security.html <
> https://solr.apache.org/security.html>
>
> Solr 4 is before Solr 7.4, so it is not affected by this problem.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Dec 11, 2021, at 8:28 PM, Reej Nayagam  wrote:
> >
> > Thank you for your reply.
> >
> > It mentions Dlog4j2 but with solr4 it is log4j1.2.17
> > Can we use this command
> >
> >   - -*Dlog4j2*.formatMsgNoLookups=true
> >
> >
> > On Sun, 12 Dec 2021 at 12:03 PM, Raveendra Yerraguntla
> >  wrote:
> >
> >>
> >>   - -Dlog4j2.formatMsgNoLookups=true
> >>
> >>
> >> restart jvm with the above param and should work.
> >>
> >>
> >>
> >>
> >>
> >>On Saturday, December 11, 2021, 09:51:54 PM EST, Reej Nayagam <
> >> reej...@gmail.com> wrote:
> >>
> >> Hi All,
> >>
> >> In production we are using solr4 which uses log4j-1.2.17.jar.
> >>
> >> Can someone say the mitigation option for solr4
> >>
> >> Thanks
> >> Reej
> >> --
> >> *Thanks,*
> >> *Reej*
> >>
> >
> > --
> > *Thanks,*
> > *Reej*
>
>

Re: Log4j vulnerability- Solr4 - urgent pls

[Reej Nayagam]

Thanks for the reply.

*REgards,*
*Reej*


On Sun, Dec 12, 2021 at 12:28 PM Rahul Goswami 
wrote:

> In case of solr4 which uses log4j-1.2.17.jar, the
> "log4j2.formatMsgNoLookups=true" system property is neither required nor
> applicable. In fact, the property was only introduced in log4j-2.10 (refer
> to the JIRA below). So not just Solr, but any Java application using 2<=
> log4j <2.10 will not be helped by this system property.
>
> https://issues.apache.org/jira/browse/LOG4J2-2109
>
> On Sat, Dec 11, 2021 at 11:04 PM Raveendra Yerraguntla
>  wrote:
>
> >
> >- -Dlog4j2.formatMsgNoLookups=true
> >
> >
> > restart jvm with the above param and should work.
> >
> >
> >
> >
> >
> > On Saturday, December 11, 2021, 09:51:54 PM EST, Reej Nayagam <
> > reej...@gmail.com> wrote:
> >
> >  Hi All,
> >
> > In production we are using solr4 which uses log4j-1.2.17.jar.
> >
> > Can someone say the mitigation option for solr4
> >
> > Thanks
> > Reej
> > --
> > *Thanks,*
> > *Reej*
> >
>