Re: Solr heap memory

2021-09-01 Thread Dominique Bejean 
Hi,

As previously said, long GC pauses should be the cause of Solr/Zookeeper
communication issues. Analyse your GC logs with gceasy.io in order to
confirm this. After this, you need to investigate what is causing so much
heap memory consumption. Maybe you will discover misconceptions in
your shema and some of your queries (facets, leading wildcards on
misconfigured fields, huge boolean queries, ...) and you will be able to
drastically divide memory requirements. Maybe you have some much data that
you will need to reconsider your sharding and infrastructure. Anyway,
increasing heap size indefinitely is not the solution and in any case never
over 31g.

Dominique


Le lun. 30 août 2021 à 15:15, HariBabu kuruva  a
écrit :

> In logs I could see this WARN.
>
> 2021-08-30 13:15:52.301 WARN  (zkCallback-12-thread-3) [c:quoteStore
> s:shard1 r:core_node6 x:quoteStore_shard1_replica_n5]
> o.a.s.c.RecoveryStrategy Stopping recovery for
> core=[quoteStore_shard1_replica_n5] coreNodeName=[core_node6]
>
> On Mon, Aug 30, 2021 at 6:43 PM HariBabu kuruva  >
> wrote:
>
> > Hi Zisis,
> >
> > Thanks for your email.
> >
> > We are suspecting the issue with one particular solr collection(or
> > store).  Wherever the replicas of that store are present that nodes are
> > going down.
> >
> > Also now that shard is in recovery mode and Leader is not elected. Could
> > you please suggest something to bring up this store.
> >
> > On Mon, Aug 30, 2021 at 1:55 PM Zisis Tachtsidis 
> > wrote:
> >
> >> My guess is that the Solr/Zookeeper communication issues are due to GC
> >> pauses. You are saying that you end up with OOM problems. High memory
> usage
> >> puts pressure on GC. Long GC pauses lead to timeouts in Solr/Zookeeper
> >> communication. We've seen that happening.
> >>
> >> First thing I'd do is to get a heap dump once the OOM is triggered and
> >> analyze that to see what is occupying the memory. Otherwise we are blind
> >> here.  Is it due to heavy indexing? Heavy querying? Both of them? Do you
> >> have customizations in the analysis chain that might generate more
> objects
> >> than usual?
> >>
> >> Zisis
> >>
> >> On Mon, 30 Aug 2021 03:44:13 -0400, Dave 
> >> wrote:
> >>
> >> > I can’t help beyond such, I don’t like solr cloud nor zookeeper, I
> will
> >> always, if I can help it, stick to standalone solr instance.
> >> >
> >> > > On Aug 30, 2021, at 3:23 AM, HariBabu kuruva <
> >> hari2708.kur...@gmail.com> wrote:
> >> > >
> >> > > Hi Dave
> >> > >
> >> > > We tried setting the memory as per your suggestions.
> >> > >
> >> > > But still I see that the solr is going down in a couple of minutes
> >> with an
> >> > > OOM error. Also in the solr logs it says below connectivity issue
> >> between
> >> > > solr and zookeeper.  Please advise.
> >> > >
> >> > > Zookeeper is running fine.
> >> > >
> >> > >
> >> > > 2021-08-30 06:24:13.070 WARN  (main-SendThread(
> >> > > lxeisprdas06.corp.equinix.com:2181)) [   ] o.a.z.ClientCnxn Client
> >> session
> >> > > timed out, have not heard from server in 65584ms for session id
> >> > > 0x119354b021b
> >> > > 2021-08-30 06:24:13.071 WARN  (main-SendThread(
> >> > > lxeisprdas06.corp.equinix.com:2181)) [   ] o.a.z.ClientCnxn Session
> >> > > 0x119354b021b for sever
> >> lxeisprdas06.corp.equinix.com/10.**.*.*:2181,
> >> > > Closing socket connection. Attempting reconnect except it is a
> >> > > SessionExpiredException. =>
> >> > > org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client
> >> session
> >> > > timed out, have not heard from server in 65584ms for session id
> >> > > 0x119354b021b
> >> > >at
> >> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1243)
> >> > > org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client
> >> session
> >> > > timed out, have not heard from server in 65584ms for session id
> >> > > 0x119354b021b
> >> > >at
> >> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1243)
> >> > > ~[zookeeper-3.6.2.jar:3.6.2]
> >> > > 2021-08-30 06:24:26.182 ERROR (qtp1198197478-540) [   ]
> >> > > o.a.s.s.PKIAuthenticationPlugin Invalid key request timestamp:
> >> > > 1630304577209 , received timestamp: 1630304666181 , TTL: 15000
> >> > > 2021-08-30 06:24:26.182 ERROR (qtp1198197478-531) [   ]
> >> > > o.a.s.s.PKIAuthenticationPlugin Invalid key request timestamp:
> >> > > 1630304527726 , received timestamp: 1630304600766 , TTL: 15000
> >> > > 2021-08-30 06:26:36.014 WARN
> >> (zkConnectionManagerCallback-13-thread-1) [
> >> > > ] o.a.s.c.c.ConnectionManager Watcher
> >> > > org.apache.solr.common.cloud.ConnectionManager@e31302e name:
> >> > > ZooKeeperConnection Watcher:zookeeper2.corp.equinix.com:2181,
> >> > > zookeeper1.corp.equinix.com:2182,zookeeper3.corp.equinix.com:2183,
> >> > > zookeeper4.corp.equinix.com:2184,zookeeper5.corp.equinix.com:2185
> >> got event
> >> > > WatchedEvent state:Disconnected type:None path:null path: null type:
> >> None
> >> > > 2021-08-30 06:26:36.014

RE: Vulnerabilities in SOLR 8.8.2

2021-09-01 Thread Narayanan, Lakshmi 
This is my monthly reminder to SOLR support groups
Please advise if the below listed vulnerabilities have been resolved in higher 
versions of SOLR
Any response to this message will be gratefully received

Thank you

Lakshmi Narayanan
Marsh & McLennan Companies
121 River Street, Hoboken,NJ-07030
201-284-3345
M: 845-300-3809
Email: lakshmi.naraya...@mmc.com

From: Narayanan, Lakshmi 
Sent: Sunday, August 01, 2021 10:12 PM
To: solr-u...@lucene.apache.org; users@solr.apache.org
Subject: RE: Vulnerabilities in SOLR 8.8.2

Hello SOLR Support team
This is my monthly check on this subject
Is someone listening out there to help me with my question below please?
Please advise
Thank you

From: Narayanan, Lakshmi 
mailto:lakshmi.naraya...@mmc.com>>
Sent: Monday, July 05, 2021 1:27 PM
To: solr-u...@lucene.apache.org; 
users@solr.apache.org
Subject: RE: Vulnerabilities in SOLR 8.8.2

Hello SOLR User Support Team
Please advise, how to address these vulnerabilities in SOLR package
This is preventing us from going live

Please advise, if this needs to be sent to any other teams within SOLR user 
support
Thank you

Lakshmi Narayanan
Marsh & McLennan Companies
121 River Street, Hoboken,NJ-07030
201-284-3345
M: 845-300-3809
Email: lakshmi.naraya...@mmc.com

From: Narayanan, Lakshmi 
mailto:lakshmi.naraya...@mmc.com>>
Sent: Monday, June 07, 2021 4:28 PM
To: solr-u...@lucene.apache.org; 
users@solr.apache.org
Subject: RE: Vulnerabilities in SOLR 8.8.2

Sending to users@solr.apache.org

Lakshmi Narayanan
Marsh & McLennan Companies
121 River Street, Hoboken,NJ-07030
201-284-3345
M: 845-300-3809
Email: lakshmi.naraya...@mmc.com

From: Narayanan, Lakshmi 
mailto:lakshmi.naraya...@mmc.com>>
Sent: Monday, June 07, 2021 3:28 PM
To: solr-u...@lucene.apache.org
Subject: Vulnerabilities in SOLR 8.8.2

Hello SOLR-User Support team
Please advise if there is resolution to the vulnerabilities listed below in 
SOLR 8.8.2
This is preventing us from using the SOLR product

I have tried to contact this mailgroup fro support before;
Please advise if there is another mailgroup I can reach for SOLR Support?

Thank you

Lakshmi Narayanan
Marsh & McLennan Companies
121 River Street, Hoboken,NJ-07030
201-284-3345
M: 845-300-3809
Email: lakshmi.naraya...@mmc.com

Vulnerability

Severity

Package

Package Version

Package Type

Package Path

URL

Fix

Stop

Grace Period Stop

Known Warn

VULNDB-180024

High

derby

10.9.1.0

java

/opt/solr-8.8.2/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar

https://mgti-dal-so-sysdig.mrshmc.com:443/secure/#/scanning/vulnerabilities/VULNDB-180024

10.14.2.0

True

False

False

VULNDB-247944

High

hadoop

3.2.0

java

/opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/hadoop-annotations-3.2.0.jar

https://mgti-dal-so-sysdig.mrshmc.com:443/secure/#/scanning/vulnerabilities/VULNDB-247944

2.10.1, 3.1.4, 3.2.2, 3.3.0

True

False

False

VULNDB-247944

High

hadoop

3.2.0

java

/opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/hadoop-auth-3.2.0.jar

https://mgti-dal-so-sysdig.mrshmc.com:443/secure/#/scanning/vulnerabilities/VULNDB-247944

2.10.1, 3.1.4, 3.2.2, 3.3.0

True

False

False

VULNDB-247944

High

hadoop

3.2.0

java

/opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/hadoop-common-3.2.0.jar

https://mgti-dal-so-sysdig.mrshmc.com:443/secure/#/scanning/vulnerabilities/VULNDB-247944

2.10.1, 3.1.4, 3.2.2, 3.3.0

True

False

False

VULNDB-247944

High

hadoop

3.2.0

java

/opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/hadoop-hdfs-client-3.2.0.jar

https://mgti-dal-so-sysdig.mrshmc.com:443/secure/#/scanning/vulnerabilities/VULNDB-247944

2.10.1, 3.1.4, 3.2.2, 3.3.0

True

False

False

VULNDB-223108

High

jackson-databind

2.4.0

java

/opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/htrace-core4-4.1.0-incubating.jar:jackson-databind

https://mgti-dal-so-sysdig.mrshmc.com:443/secure/#/scanning/vulnerabilities/VULNDB-223108

2.8.11.5, 2.9.10.3

True

False

False

VULNDB-214563

High

jackson-databind

2.4.0

java

/opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/htrace-core4-4.1.0-incubating.jar:jackson-databind

https://mgti-dal-so-sysdig.mrshmc.com:443/secure/#/scanning/vulnerabilities/VULNDB-214563

2.10.0, 2.9.10.1

True

False

False





From: Narayanan, Lakshmi 
mailto:lakshmi.naraya...@mmc.com>>
Sent: Friday, December 11, 2020 11:50 AM
To: solr-u...@lucene.apache.org
Subject: FW: Vulnerabilities in SOLR 8.6.2

Can anyone please advise?
Who else should be notified to get some guidance on this please??

Lakshmi Narayanan
Marsh & McLennan Companies
121 River Street, Hoboken,NJ-07030
201-284-3345
M: 845-300-3809

Re: Vulnerabilities in SOLR 8.8.2

2021-09-01 Thread Shawn Heisey 

On 9/1/2021 8:25 AM, Narayanan, Lakshmi wrote:

This is my monthly reminder to SOLR support groups
Please advise if the below listed vulnerabilities have been resolved in higher 
versions of SOLR
Any response to this message will be gratefully received


The vast majority of any vulnerabilities will be impossible to exploit 
if you follow one of the most basic security steps:  Make sure that Solr 
is not accessible to the outside world.  At the network and/or OS level, 
make sure that only the IP addresses of people and applications that 
need Solr are able to access whatever port Solr is listening on.



/opt/solr-8.8.2/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar


Whatever the vulnerability is here, it can only be a problem if you 
actually use the derby database with the dataimport handler.



/opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/hadoop-annotations-3.2.0.jar
/opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/hadoop-auth-3.2.0.jar
/opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/hadoop-common-3.2.0.jar
/opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/hadoop-hdfs-client-3.2.0.jar
/opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/htrace-core4-4.1.0-incubating.jar:jackson-databind
/opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/htrace-core4-4.1.0-incubating.jar:jackson-databind


Are you using the HDFS filesystem support in Solr?  If you're not, then 
these jars are not used and you won't need to worry about it.


I'll say the first thing I said again:  The vast majority of any 
vulnerabilities will be impossible to exploit if you follow one of the 
most basic security steps:  Make sure that Solr is not accessible to the 
outside world.  At the network and/or OS level, make sure that only the 
IP addresses of people and applications that need Solr are able to 
access whatever port Solr is listening on.


If you can't trust your own people, that is an internal security issue 
for your organization, and the Solr project cannot help with it.


Thanks,
Shawn

Re: Vulnerabilities in SOLR 8.8.2

2021-09-01 Thread Dave 
Agreed. Solr should always have a front end to interface with the server 
itself.  I don’t think I’ve ever seen a situation where it was accessible 
outside of the internal network.  Not to mention it gives you an extra layer to 
add parameters or clean user input. Raw solr is for the developers, that make 
the interface to the user input 

> On Sep 1, 2021, at 11:28 AM, Shawn Heisey  wrote:
> 
> On 9/1/2021 8:25 AM, Narayanan, Lakshmi wrote:
>> This is my monthly reminder to SOLR support groups
>> Please advise if the below listed vulnerabilities have been resolved in 
>> higher versions of SOLR
>> Any response to this message will be gratefully received
> 
> The vast majority of any vulnerabilities will be impossible to exploit if you 
> follow one of the most basic security steps:  Make sure that Solr is not 
> accessible to the outside world.  At the network and/or OS level, make sure 
> that only the IP addresses of people and applications that need Solr are able 
> to access whatever port Solr is listening on.
> 
>> /opt/solr-8.8.2/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar
> 
> Whatever the vulnerability is here, it can only be a problem if you actually 
> use the derby database with the dataimport handler.
> 
>> /opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/hadoop-annotations-3.2.0.jar
>> /opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/hadoop-auth-3.2.0.jar
>> /opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/hadoop-common-3.2.0.jar
>> /opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/hadoop-hdfs-client-3.2.0.jar
>> /opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/htrace-core4-4.1.0-incubating.jar:jackson-databind
>> /opt/solr-8.8.2/server/solr-webapp/webapp/WEB-INF/lib/htrace-core4-4.1.0-incubating.jar:jackson-databind
> 
> Are you using the HDFS filesystem support in Solr?  If you're not, then these 
> jars are not used and you won't need to worry about it.
> 
> I'll say the first thing I said again:  The vast majority of any 
> vulnerabilities will be impossible to exploit if you follow one of the most 
> basic security steps:  Make sure that Solr is not accessible to the outside 
> world.  At the network and/or OS level, make sure that only the IP addresses 
> of people and applications that need Solr are able to access whatever port 
> Solr is listening on.
> 
> If you can't trust your own people, that is an internal security issue for 
> your organization, and the Solr project cannot help with it.
> 
> Thanks,
> Shawn
>

How to get rid of this Warning "WARN (qtp1533985074-57) [ ] o.a.h.s.a.u.KerberosName auth_to_local rule mechanism not set.Using default of hadoop"

2021-09-01 Thread jrballesteros05 
Hello everybody, this is my first time asking for support here. I normally 
search in the web and look in the forums list but I don't know how to do it 
here, I am very clumsy with mailing list, so If this post is duplicated or 
someone else has asked this question before I would appreciate if you point me 
where to look.

I configured Solr authentication this guide:

https://solr.apache.org/guide/8_9/kerberos-authentication-plugin.html

Everything is working OK, I just receive this warning message so often:

2021-09-01 20:29:46.789 WARN  (qtp1533985074-61) [   ] o.a.h.s.a.u.KerberosName 
auth_to_local rule mechanism not set.Using default of hadoop

I don't know what to do to get rid of this. I personally want to make a right 
configuration in the right file instead of just disabling the warning.

I just configured the "solr.kerberos.name .rules" in 
the solr.in.sh  but it seems to be ignored.  I don't know if 
I have to make an extra configuration in the FreeIPA or maybe I am missing 
another configuration file. As I understand this warning should be if I use 
HDFS or Hadoop Authentication but this is not the case.

This is the content of my solr.in.sh  file:

SOLR_PID_DIR="/opt/var/solr"
SOLR_HOME="/opt/var/solr/data"
LOG4J_PROPS="/opt/var/solr/log4j2.xml"
SOLR_LOGS_DIR="/opt/var/solr/logs"
SOLR_PORT="8983"
SOLR_HEAP="6g"
SOLR_HOST="sa3secglbsolr01.a3sec.local"
ZK_HOST="sa3secglbzkpt01.a3sec.local:2181,sa3secglbzkpt02.a3sec.local:2181,sa3secglbzkpt03.a3sec.local:2181/solr"

# Settings for ZK ACL
SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider
 \
  
-DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider
 \
  -DzkDigestUsername=admin-user -DzkDigestPassword=anypassword \
  -DzkDigestReadonlyUsername=readonly-user 
-DzkDigestReadonlyPassword=anypassword"
SOLR_OPTS="$SOLR_OPTS $SOLR_ZK_CREDS_AND_ACLS"

# Enables HTTPS. It is implicitly true if you set SOLR_SSL_KEY_STORE. Use this 
config
# to enable https module with custom jetty configuration.
SOLR_SSL_ENABLED=true
# Uncomment to set SSL-related system properties
# Be sure to update the paths to the correct keystore for your environment
SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.p12
SOLR_SSL_KEY_STORE_PASSWORD=
SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.p12
SOLR_SSL_TRUST_STORE_PASSWORD=
# Require clients to authenticate
SOLR_SSL_NEED_CLIENT_AUTH=false
# Enable clients to authenticate (but not require)
SOLR_SSL_WANT_CLIENT_AUTH=false
# SSL Certificates contain host/ip "peer name" information that is validated by 
default. Setting
# this to false can be useful to disable these checks when re-using a 
certificate on many hosts
SOLR_SSL_CHECK_PEER_NAME=true

KERBEROS_RULE="RULE:[1:\$1@\$0](.*A3SEC.LOCAL)s/@.*//"
SOLR_AUTH_TYPE="kerberos"
SOLR_AUTHENTICATION_OPTS="-Djava.security 
.auth.login.config=/home/debian/jaas-client.conf 
-Dsolr.kerberos.cookie.domain=sa3secglbsolr01.a3sec.local 
-Dsolr.kerberos.cookie.portaware=true 
-Dsolr.kerberos.principal=HTTP/sa3secglbsolr01.a3sec.local@A3SEC.LOCAL 
-Dsolr.kerberos.keytab=/home/debian/sa3secglbsolr01.keytab -Dsolr.kerberos.name 
.rules=$KERBEROS_RULE"

In the worst case, is there a way to disable that specific Warning?

Best regards.

Re: How to get rid of this Warning "WARN (qtp1533985074-57) [ ] o.a.h.s.a.u.KerberosName auth_to_local rule mechanism not set.Using default of hadoop"

2021-09-01 Thread Shawn Heisey 

On 9/1/2021 2:48 PM, jrballestero...@tutanota.com.INVALID wrote:

Everything is working OK, I just receive this warning message so often:

2021-09-01 20:29:46.789 WARN  (qtp1533985074-61) [   ] o.a.h.s.a.u.KerberosName 
auth_to_local rule mechanism not set.Using default of hadoop



As I understand this warning should be if I use HDFS or Hadoop Authentication 
but this is not the case.


Since you're not using HDFS, you don't need logs from hadoop.  In your 
log4j2.xml file, you probably have this line in the  Section:


    

If you simply change "warn" to "off" on that line, that should suppress 
all logging from hadoop classes, and give you the results you're after.


If you don't have that line, download the latest Solr and find 
log4j2.xml in the archive, see how it is constructed, and then adjust 
your own log4j2.xml accordingly.


We really should track down why a hadoop class is getting involved when 
you are not using HDFS, but that's probably going to take a lot of 
effort and I don't even know where to begin.


(note that if you are running a Solr version before 7.4, your config 
will not be in log4j2.xml, it will probably be in log4j.properties ... 
and if you're running a REALLY old version, then the log config could be 
in an unknown location.)


Thanks,
Shawn

Re: How to get rid of this Warning "WARN (qtp1533985074-57) [ ] o.a.h.s.a.u.KerberosName auth_to_local rule mechanism not set.Using default of hadoop"

2021-09-01 Thread Chris Hostetter 

: Everything is working OK, I just receive this warning message so often:
: 
: 2021-09-01 20:29:46.789 WARN  (qtp1533985074-61) [   ] 
o.a.h.s.a.u.KerberosName auth_to_local rule mechanism not set.Using default of 
hadoop

...

: file. As I understand this warning should be if I use HDFS or Hadoop 
: Authentication but this is not the case.

IIUC, the Solr KerberosPlugin uses hadoop Kerberos utility classes 
under the covers, even when you aren't explicitly using hadoop based 
authentication ... i gather that one/some of those utilities make 
assumptions about how they are being used.

I don't know that there is anything you personally can change about how 
solr is configured to prevent this WARN from happening ... from poking 
around the KerberosName.java code, i gather that somewhere in the Solr 
KerberosPlugin lifecycle, solr should be calling 
KerberosName.setRuleMechanism(...) (or calling some other hadoop kerberos 
helper util in such a way that it calls that method)

Can you please file a jira pointing out this warning and this email 
thread? ... in the meantime i think the only thing you can do is disable 
the WARN level from the KerberosName logger in your log4j config.


-Hoss
http://www.lucidworks.com/

Solr Windows Service / Running solr in the background

2021-09-01 Thread Reej M 
Hi All,

For now we are running solr cloud starting it from the command prompt as below 
from the 2 servers for starting the 2 nodes

Server 1 & Server 2
bin\solr start -cloud -p 3883 -s “server\solr\cloud\node\solr”
bin\solr start -cloud -p 9883 -s “server\solr\cloud\node\solr”

But if we close the command prompt server will obviously go down, is there any 
way we can have this running in the background.

We tried using shell script, but we can run it as a standalone service, but is 
not working when we pass the commands to start as a cloud service.
We followed  http://lets-share.senktas.net/2017/11/solr-as-a-service.html 
 for running as 
windows service.

Any one has tried any options for starting solr cloud as windows service, do 
share your thoughts.

Thanks
Reej