Solr operator and side car container
Hi, Is there any facility in the Solr operator for adding a sidecar container that runs along side the solr container? I see there is an initContainer but I don't see anything for adding another container. Thanks, Joel
Re: Implementing Authentication for Solr Indexing and Querying in a Spring Boot Project
Hello all, Do we have any update on best practices related to above mentioned challenges? *Thanks & Regards,* *Uday Kumar* On Mon, Dec 11, 2023 at 2:18 PM Uday Kumar wrote: > Hello all, > > We are working on a project which involves two processes, indexing > documents into Solr cloud and querying from Solr cloud. > > We are trying to implement authentication at the Solr Cloud using the > Basic Authentication *[with username and password]* plugin to ensure that > only authenticated users can access. > > We have encountered a couple of challenges regarding the implementation of > authentication at both the indexing and querying processes. Specifically: > > *1. Authentication for Indexing:* > > We are currently using the SolrJ client for indexing documents into the > Solr index. However, with authentication enabled at Solr, we are uncertain > about the best practices for implementing authentication within the SolrJ > client during the indexing process. Could you provide guidance on how to > seamlessly integrate authentication into the SolrJ client for secure > document indexing? > > *2. Authentication for Querying:* > > On the querying side, we are using the `HttpURLConnection` package for > making requests to Solr Cloud. Given the authentication requirements, I am > seeking advice on how to properly authenticate and handle queries securely > using `HttpURLConnection`. > > *3. Secure Storage of User Credentials in Spring Boot:* > > Additionally, we are exploring ways to securely store Solr Cloud > credentials (username and password) within our Spring Boot application > instead of raw text in properties file. Could you provide recommendations > or best practices for securely storing and retrieving Solr Cloud > credentials in a Spring Boot application? > > *Thanks & Regards,* > *Uday Kumar* >
Re: Java Upgrade process
On 12/10/23 20:36, Jim Morgan wrote: Now we're looking at java. Its currently running on openjdk-8. I've tried upgrading to openjdk-11 and openjdk-13. The process, as this is Ubuntu is basically. Don't use Java 13. It is a non-LTS release. Non-LTS releases are only supported for 6 months, so that version has been out of support for more than three years -- support ended in March 2020. Java 11 is a good solid choice. Java 17 seems to perform a little better than 11 on Solr 9.x, but I haven't actually measured it. Java 8, 11, 17, and 21 are the current LTS releases. https://en.wikipedia.org/wiki/Java_version_history I personally wouldn't trust Solr 8.x with Java 17, as that's a jump of 9 major Java versions ... but others have done it and I haven't seen reports of problems. I use Java 17 with Solr 9.x on Ubuntu. With Java 11 or later, I think it would be a good idea to switch garbage collection to ZGC. I have this in my /etc/default/solr.in.sh: GC_TUNE=" \ -XX:+UnlockExperimentalVMOptions \ -XX:+UseZGC \ -XX:+ParallelRefProcEnabled \ -XX:+ExplicitGCInvokesConcurrent \ -XX:+AlwaysPreTouch \ -XX:+UseNUMA \ " Side note: You should be using update-java-alternatives instead of update-alternatives. It will update ALL of the binaries for java, not just /usr/bin/java. Showing available options: elyograg@smeagol:~$ update-java-alternatives -l java-1.11.0-openjdk-amd64 /usr/lib/jvm/java-1.11.0-openjdk-amd64 java-1.17.0-openjdk-amd64 1711 /usr/lib/jvm/java-1.17.0-openjdk-amd64 java-1.8.0-openjdk-amd64 1081 /usr/lib/jvm/java-1.8.0-openjdk-amd64 You would then follow that with something like: update-java-alternatives -s java-1.11.0-openjdk-amd64 If Solr will not start with Java 11, look in solr.log and solr--console.log for info about why it failed. Thanks, Shawn
Re: Solr connection refused error
On 12/11/23 21:25, Anuj Bhargava wrote: http://xxx.xxx.xx.xxx:8983/solr/#/ I often get the following error - This site can’t be reached xxx.xxx.xx.xxx refused to connect. Try: Checking the connection Checking the proxy and the firewall ERR_CONNECTION_REFUSED 8983 port is open There's not a lot of info here. Starting in version 9.0.0, Solr only listens on localhost by default. To listen on all interfaces, set the SOLR_JETTY_HOST environment variable to 0.0.0.0 in the include script. Thanks, Shawn
Re: Java Upgrade process
That's the console output, where the start script is waiting for the port to come up. what does the actual solr log file (written by the java process) say? https://solr.apache.org/guide/solr/latest/deployment-guide/taking-solr-to-production.html#log-settings : Date: Tue, 12 Dec 2023 10:57:37 +0800 : From: Jim Morgan : Reply-To: users@solr.apache.org : To: users@solr.apache.org : Subject: Re: Java Upgrade process : : Fair point Chris, I should have included log info in the original post. : So in the solr logs, nothing at all. I think because solr doesn't get to : the point of starting. : In syslog I have : - : Dec 11 06:35:08 server solr[6822]: Waiting up to 180 seconds to see Solr : running on port 8983 [|] #010#010#010#010#010#010 [/] : #010#010#010#010#010#010 [|] #010#010#010#010#010#010 [/] : (lots and lots of these removed) : 010#010#010 [-] #010#010#010#010#010 : Dec 11 06:35:08 server solr[6897]: Still not seeing Solr listening on 8983 : after 180 seconds! : - : There was a warning about Ulimits, which doesn't seem to occur under Java 8: : Dec 11 06:30:32 solr-dev solr[6266]: *** [WARN] *** Your open file limit is : currently 1024. : Dec 11 06:30:32 solr-dev solr[6266]: It should be set to 65000 to avoid : operational disruption. : Dec 11 06:30:32 solr-dev solr[6266]: If you no longer wish to see this : warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh : But I set that in /etc/defaults/solr.in.sh and it didn't help. : : Jim : : On Tue, Dec 12, 2023 at 8:10 AM Chris Hostetter : wrote: : : > : > can you be more specific about " It just waits and eventually fails after : > about 2 minutes." ... what message does it print to the consolr? what : > does the solr log file say? : > : > : > : > : Date: Mon, 11 Dec 2023 11:36:15 +0800 : > : From: Jim Morgan : > : Reply-To: users@solr.apache.org : > : To: users@solr.apache.org : > : Subject: Java Upgrade process : > : : > : A few months back, I had a question about upgrading Solr. I was trying to : > : upgrade from 8.5 to 9.3. That introduced a few errors into our website : > : search that we were not able to resolve, so we've figured we'll just : > : upgrade to the latest 8.x version, ie. 8.11. : > : : > : We've successfully upgraded the underlying OS, and we're happy 8.11 : > returns : > : the same search results as 8.5. So far so good. : > : : > : Now we're looking at java. Its currently running on openjdk-8. I've tried : > : upgrading to openjdk-11 and openjdk-13. The process, as this is Ubuntu is : > : basically. : > : apt install openjdk-13-jdk : > : update-alternatives --list java : > : update-alternatives --set java : > /usr/lib/jvm/java-13-openjdk-amd64/bin/java : > : On doing that, solr will fail to start. It just waits and eventually : > fails : > : after about 2 minutes. So I can do : > : update-alternatives --set java : > : /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java : > : To set it back to java 8 and it runs again. : > : : > : I've tried regenerating the init script with the new java, but the : > scripts : > : are identical. I've tried rebooting the server with the new java active : > in : > : case its an environment thing. But to no avail. : > : : > : Any ideas what's going on here? : > : : > : I notice java 8 is still getting updates, so maybe the path of least : > : resistance is to stay on that. : > : : > : Jim : > : : > : > -Hoss : > http://www.lucidworks.com/ : > : -Hoss http://www.lucidworks.com/
Re: Solr 9.4 - Switching _default configset to classic schema issue
Hi,
just stumbled upon this: This looks like open bug
https://issues.apache.org/jira/browse/SOLR-16203
The workaround would be to configure StopFilterFactory and similar
analysis factories with a class attribute and not the name attribute.
For example, use class="solr.StopFilterFactory" instead of name="stop"
Cheers
Andreas
Dominique Bejean schrieb am 10.11.23 um 00:16:
Hi,
I created a default-classic configset by :
* copying the _default configset
* rename managed-schema.xml file to schema.xml
* adddeclaration in
solrconfig.xml
I can successfully create a collection
When I test a query analysis in Admin UI with the field type "text
general", the following error occurs
2023-11-09 23:04:10.320 INFO (qtp1677458155-33) [c:boamp s:shard1
r:core_node5 x:boamp_shard1_replica_n2] o.a.s.c.S.Request webapp=/solr
path=/analysis/field params={analysis.fieldvalue=the
dog&analysis.showmatch=true&verbose_output=1&wt=json&_=1699571050222&analysis.fieldtype=text_general}
status=500 QTime=0
2023-11-09 23:04:10.320 ERROR (qtp1677458155-33) [c:boamp s:shard1
r:core_node5 x:boamp_shard1_replica_n2] o.a.s.s.HttpSolrCall 500 Exception
=> java.lang.NullPointerException: stopWords
at java.base/java.util.Objects.requireNonNull(Unknown Source)
java.lang.NullPointerException: stopWords
at java.util.Objects.requireNonNull(Unknown Source) ~[?:?]
at org.apache.lucene.analysis.StopFilter.(StopFilter.java:39)
~[?:?]
at org.apache.lucene.analysis.core.StopFilter.(StopFilter.java:43)
~[?:?]
at
org.apache.lucene.analysis.core.StopFilterFactory.create(StopFilterFactory.java:91)
~[?:?]
at
org.apache.solr.handler.AnalysisRequestHandlerBase.analyzeValue(AnalysisRequestHandlerBase.java:141)
~[?:?]
at
org.apache.solr.handler.FieldAnalysisRequestHandler.analyzeValues(FieldAnalysisRequestHandler.java:229)
~[?:?]
at
org.apache.solr.handler.FieldAnalysisRequestHandler.handleAnalysisRequest(FieldAnalysisRequestHandler.java:186)
~[?:?]
at
org.apache.solr.handler.FieldAnalysisRequestHandler.doAnalysis(FieldAnalysisRequestHandler.java:103)
~[?:?]
...
Same issue with a filter like "elision"
2023-11-09 23:15:31.143 ERROR (qtp1677458155-33) [c:boamp s:shard1
r:core_node5 x:boamp_shard1_replica_n2] o.a.s.s.HttpSolrCall 500 Exception
=> java.lang.NullPointerException: articles
at java.base/java.util.Objects.requireNonNull(Unknown Source)
java.lang.NullPointerException: articles
at java.util.Objects.requireNonNull(Unknown Source) ~[?:?]
at
org.apache.lucene.analysis.util.ElisionFilter.(ElisionFilter.java:44)
~[?:?]
at
org.apache.lucene.analysis.util.ElisionFilterFactory.create(ElisionFilterFactory.java:79)
~[?:?]
It looks like resource files are not correctly loaded with
ClassicIndexSchemaFactory enabled with Solr 9.4 (with solr 9.3 too).
It works perfectly with Solr 8.11.2
Any suggestions ?
Regards
Dominique
Re: Implementing Authentication for Solr Indexing and Querying in a Spring Boot Project
On 12/11/23 01:48, Uday Kumar wrote: *1. Authentication for Indexing:* We are currently using the SolrJ client for indexing documents into the Solr index. However, with authentication enabled at Solr, we are uncertain about the best practices for implementing authentication within the SolrJ client during the indexing process. Could you provide guidance on how to seamlessly integrate authentication into the SolrJ client for secure document indexing? *2. Authentication for Querying:* On the querying side, we are using the `HttpURLConnection` package for making requests to Solr Cloud. Given the authentication requirements, I am seeking advice on how to properly authenticate and handle queries securely using `HttpURLConnection`. You can use a SolrJ client for both querying and indexing, there is no reason to use Java's HTTP. With Java's client, you have to handle the extraction of data from the response by parsing JSON or XML. SolrJ does the bulk of that work for you, and uses a more compact binary format on the wire rather than JSON. Here is some code for your perusal: https://paste.elyograg.org/view/e5cce5db Lines 26 and 27 are related to other parts of the overall program and don't contribute to client creation. A summary of what that code does: * create a builder for Http2SolrClient for talking to the source cluster. * create a builder for Http2SolrClient as a "helper" client for the target cluster. * add authentication info to those two client builders. * create a builder for CloudHttp2SolrClient for talking to the target cluster. * build the helper client for the target. * build the source client. * build the final target client. *3. Secure Storage of User Credentials in Spring Boot:* Additionally, we are exploring ways to securely store Solr Cloud credentials (username and password) within our Spring Boot application instead of raw text in properties file. Could you provide recommendations or best practices for securely storing and retrieving Solr Cloud credentials in a Spring Boot application? You'll want to discuss that with Spring or in a general forum for Java. In general, my opinion is that there is no real value to trying to obscure credentials beyond file permissions ... if someone manages to achieve admin capability on the server, they will be almost certainly be able to just use whatever mechanisms are already in place and gain access. Thanks, Shawn
Compatibility of Solr Cloud v8.11.1 with Zookeeper v3.8.1
Hello, We are currently using Solr cloud with solr v8.11.1 and zookeeper v3.6.1. We want to upgrade zookeeper to v3.8.1. Any compatibility issues between zk v3.8.1 with solr v8.11.1. We know the recommendation is to use Solr v9.2. However, we wonder if this cause combability with our Drupal 10 site with the customization we have in place. Please advise. Thanks
Re: Solr 9.4 - Switching _default configset to classic schema issue
Would you be interested in opening a PR for this? I’d love to see what that
looks like!
> On Dec 12, 2023, at 3:00 PM, Andreas Hubold
> wrote:
>
> Hi,
>
> just stumbled upon this: This looks like open bug
> https://issues.apache.org/jira/browse/SOLR-16203
>
> The workaround would be to configure StopFilterFactory and similar analysis
> factories with a class attribute and not the name attribute.
> For example, use class="solr.StopFilterFactory" instead of name="stop"
>
> Cheers
> Andreas
>
> Dominique Bejean schrieb am 10.11.23 um 00:16:
>> Hi,
>>
>> I created a default-classic configset by :
>> * copying the _default configset
>> * rename managed-schema.xml file to schema.xml
>> * adddeclaration in
>> solrconfig.xml
>>
>> I can successfully create a collection
>> When I test a query analysis in Admin UI with the field type "text
>> general", the following error occurs
>>
>> 2023-11-09 23:04:10.320 INFO (qtp1677458155-33) [c:boamp s:shard1
>> r:core_node5 x:boamp_shard1_replica_n2] o.a.s.c.S.Request webapp=/solr
>> path=/analysis/field params={analysis.fieldvalue=the
>> dog&analysis.showmatch=true&verbose_output=1&wt=json&_=1699571050222&analysis.fieldtype=text_general}
>> status=500 QTime=0
>> 2023-11-09 23:04:10.320 ERROR (qtp1677458155-33) [c:boamp s:shard1
>> r:core_node5 x:boamp_shard1_replica_n2] o.a.s.s.HttpSolrCall 500 Exception
>> => java.lang.NullPointerException: stopWords
>>at java.base/java.util.Objects.requireNonNull(Unknown Source)
>> java.lang.NullPointerException: stopWords
>>at java.util.Objects.requireNonNull(Unknown Source) ~[?:?]
>>at org.apache.lucene.analysis.StopFilter.(StopFilter.java:39)
>> ~[?:?]
>>at org.apache.lucene.analysis.core.StopFilter.(StopFilter.java:43)
>> ~[?:?]
>>at
>> org.apache.lucene.analysis.core.StopFilterFactory.create(StopFilterFactory.java:91)
>> ~[?:?]
>>at
>> org.apache.solr.handler.AnalysisRequestHandlerBase.analyzeValue(AnalysisRequestHandlerBase.java:141)
>> ~[?:?]
>>at
>> org.apache.solr.handler.FieldAnalysisRequestHandler.analyzeValues(FieldAnalysisRequestHandler.java:229)
>> ~[?:?]
>>at
>> org.apache.solr.handler.FieldAnalysisRequestHandler.handleAnalysisRequest(FieldAnalysisRequestHandler.java:186)
>> ~[?:?]
>>at
>> org.apache.solr.handler.FieldAnalysisRequestHandler.doAnalysis(FieldAnalysisRequestHandler.java:103)
>> ~[?:?]
>>...
>>
>> Same issue with a filter like "elision"
>>
>> 2023-11-09 23:15:31.143 ERROR (qtp1677458155-33) [c:boamp s:shard1
>> r:core_node5 x:boamp_shard1_replica_n2] o.a.s.s.HttpSolrCall 500 Exception
>> => java.lang.NullPointerException: articles
>>at java.base/java.util.Objects.requireNonNull(Unknown Source)
>> java.lang.NullPointerException: articles
>>at java.util.Objects.requireNonNull(Unknown Source) ~[?:?]
>>at
>> org.apache.lucene.analysis.util.ElisionFilter.(ElisionFilter.java:44)
>> ~[?:?]
>>at
>> org.apache.lucene.analysis.util.ElisionFilterFactory.create(ElisionFilterFactory.java:79)
>> ~[?:?]
>>
>> It looks like resource files are not correctly loaded with
>> ClassicIndexSchemaFactory enabled with Solr 9.4 (with solr 9.3 too).
>>
>> It works perfectly with Solr 8.11.2
>>
>> Any suggestions ?
>>
>> Regards
>>
>> Dominique
>>
>
___
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
http://www.opensourceconnections.com
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
This e-mail and all contents, including attachments, is considered to be
Company Confidential unless explicitly stated otherwise, regardless of whether
attachments are marked as such.
Re: Solr connection refused error
The solr stops too often and I have to restart it. Is there a script which checks if solr is running and if not it starts automatically. Regards, Anuj On Tue, 12 Dec 2023 at 23:29, Shawn Heisey wrote: > On 12/11/23 21:25, Anuj Bhargava wrote: > > http://xxx.xxx.xx.xxx:8983/solr/#/ > > > > I often get the following error - > > > > This site can’t be reached xxx.xxx.xx.xxx refused to connect. > > Try: > > > > Checking the connection > > Checking the proxy and the firewall > > ERR_CONNECTION_REFUSED > > > > 8983 port is open > > There's not a lot of info here. Starting in version 9.0.0, Solr only > listens on localhost by default. To listen on all interfaces, set the > SOLR_JETTY_HOST environment variable to 0.0.0.0 in the include script. > > Thanks, > Shawn > >
Re: Solr connection refused error
On Wed, Dec 13, 2023 at 11:03 AM Anuj Bhargava wrote: > The solr stops too often and I have to restart it. Is there a script which > checks if solr is running and if not it starts automatically. > You need to find why Solr is stopping too often. Restarting it is not a graceful option. > > Regards, > > Anuj > > On Tue, 12 Dec 2023 at 23:29, Shawn Heisey > wrote: > > > On 12/11/23 21:25, Anuj Bhargava wrote: > > > http://xxx.xxx.xx.xxx:8983/solr/#/ > > > > > > I often get the following error - > > > > > > This site can’t be reached xxx.xxx.xx.xxx refused to connect. > > > Try: > > > > > > Checking the connection > > > Checking the proxy and the firewall > > > ERR_CONNECTION_REFUSED > > > > > > 8983 port is open > > > > There's not a lot of info here. Starting in version 9.0.0, Solr only > > listens on localhost by default. To listen on all interfaces, set the > > SOLR_JETTY_HOST environment variable to 0.0.0.0 in the include script. > > > > Thanks, > > Shawn > > > > > Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi" +91 73500 12833 deic...@gmail.com LinkedIn: www.linkedin.com/in/deicool Upwork: https://www.upwork.com/freelancers/~0150257dd6dfd15b4b "Plant a Tree, Go Green" Make In India : http://www.makeinindia.com/home
Numfound Missmatch with Implicit collection
Hi, I'm using solr 9.1.1 and facing issue in numFound with Implicit collection. observation is when I send query to implicit collection with different rows count, it returns different numFound each time. And with the same query on Composite collection it works fine and returns correct numFound. Regards, Bilal Qureshi
Re: Implementing Authentication for Solr Indexing and Querying in a Spring Boot Project
Thank You Shawn,will go through suggestions provided! *Thanks & Regards,* *Uday Kumar* On Wed, Dec 13, 2023 at 1:35 AM Shawn Heisey wrote: > On 12/11/23 01:48, Uday Kumar wrote: > > *1. Authentication for Indexing:* > > > > We are currently using the SolrJ client for indexing documents into the > > Solr index. However, with authentication enabled at Solr, we are > uncertain > > about the best practices for implementing authentication within the SolrJ > > client during the indexing process. Could you provide guidance on how to > > seamlessly integrate authentication into the SolrJ client for secure > > document indexing? > > > > *2. Authentication for Querying:* > > > > On the querying side, we are using the `HttpURLConnection` package for > > making requests to Solr Cloud. Given the authentication requirements, I > am > > seeking advice on how to properly authenticate and handle queries > securely > > using `HttpURLConnection`. > > You can use a SolrJ client for both querying and indexing, there is no > reason to use Java's HTTP. With Java's client, you have to handle the > extraction of data from the response by parsing JSON or XML. SolrJ does > the bulk of that work for you, and uses a more compact binary format on > the wire rather than JSON. > > Here is some code for your perusal: > > https://paste.elyograg.org/view/e5cce5db > > Lines 26 and 27 are related to other parts of the overall program and > don't contribute to client creation. A summary of what that code does: > > * create a builder for Http2SolrClient for talking to the source cluster. > * create a builder for Http2SolrClient as a "helper" client for the > target cluster. > * add authentication info to those two client builders. > * create a builder for CloudHttp2SolrClient for talking to the target > cluster. > * build the helper client for the target. > * build the source client. > * build the final target client. > > > *3. Secure Storage of User Credentials in Spring Boot:* > > > > Additionally, we are exploring ways to securely store Solr Cloud > > credentials (username and password) within our Spring Boot application > > instead of raw text in properties file. Could you provide recommendations > > or best practices for securely storing and retrieving Solr Cloud > > credentials in a Spring Boot application? > > You'll want to discuss that with Spring or in a general forum for Java. > > In general, my opinion is that there is no real value to trying to > obscure credentials beyond file permissions ... if someone manages to > achieve admin capability on the server, they will be almost certainly be > able to just use whatever mechanisms are already in place and gain access. > > Thanks, > Shawn > >
