Re: HTTPSolrClient - help required

[Reej M]

Oh ok Walter. 
For the moment, we too cannot update to cloudsolrclient, and we are trying to 
find a way to resume the connections for now, and later work on the code 
cleanup. Thanks

> On 30 Jun 2021, at 12:49 AM, Walter Underwood  wrote:
> 
> CloudSolrClient is not an absolute requirement for a Solr Cloud cluster. 
> 
> We use regular HTTPSolrClient sending all requests to the load balancer.
> Actually, we use a separate load balancer for indexing, to keep the monitoring
> separate and to set different timeouts than for queries.
> 
> This setup is simple and fast. With our biggest cluster, we index about a 
> half million documents per minute.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
>> On Jun 29, 2021, at 9:37 AM, Reej Nayagam  wrote:
>> 
>> Thanks Shawn & Vicenzo. Will check it out and change accordingly. Thanks
>> again Shawn for your clear explanation.
>> 
>> Regards
>> Reej
>> 
>> 
>> On Tue, 29 Jun 2021 at 9:47 PM, Vincenzo D'Amore  wrote:
>> 
>>> Right, you should always use CloudSolrClient as a singleton.
>>> To be honest I'm used to reuse a CloudSolrClient instance for each
>>> collection/index.
>>> 
>>> On Tue, Jun 29, 2021 at 3:12 PM Shawn Heisey  wrote:
>>> 
 On 6/29/2021 6:43 AM, Reej Nayagam wrote:
> Hi Vincenzo Yes we are using cloud and initial solr version was 4.10.4
> and we upgraded the jars alone to 8.8.2 now in the application side
> connecting to solr Server to fix some vulnerability. As we have
> upgraded the jars we changed httpsolrserver connection to
> httpsolrclient and we guess there is some connection leak and wanted
> to check if we need to close it or it is being handled internally.
> Singleton not sure if I can use as the base URL changes depending on
> the leader.
 
 If you're running SolrCloud, you should be using CloudSolrClient, not
 HttpSolrClient.  The cloud client talks to zookeeper, so it is always
 aware of the cluster state -- it will be aware of down servers and new
 servers, without recreating or reinitializing the client.  And it will
 be aware of changes instantly -- because that information is coordinated
 in zookeeper.
 
 You can use a single client object for multiple collections.  All of the
 methods that execute requests should have a version where you can pass
 it the name of the collection/core you want to operate on. For
 CloudSolrClient, you point it at all your ZK servers and it figures out
 the Solr server URLs  from the clusterstate in ZK.  For HttpSolrClient,
 you just leave the collection name off of the base URL --
 "http://server.example.com:8983/solr"; is an example URL.
 
 As was mentioned, you should create a client during program startup and
 then use it to handle all requests for the life of the program. It
 should manage connections and close them after receiving data, with no
 coding needed from the developer (you).  If you close a SolrClient
 object, it will not function after that.  If you're having connections
 stay open, then either you're running a Solr or SolrJ version with bugs,
 or there is something wrong with your networking.
 
 It shouldn't be necessary to ever close a SolrClient object, unless you
 create a new one every time your program talks to Solr.   Which you
 shouldn't do.
 
 
 Thanks,
 Shawn
 
 
>>> 
>>> --
>>> Vincenzo D'Amore
>>> 
>> -- 
>> *Thanks,*
>> *Reej*
> 

Re: HTTPSolrClient - help required [Singleton Recommended?]

[Reej M]

Hi Shawn / Team ,
Need a suggestion on using the cloudsolrclient.
In our application, we have few cores which will be indexing every few minutes 
(starting from 15 mins intervall and searching will also be done by the users 
at the same time. Is it recommended to maintain a single cloudsolrclient 
throughout the application, something like a singleton? Im afraid if in a multi 
threaded env one thread shouldn’t hold the processing until the other 
completes. Kindly advise.

Thanks
Reej

> On 30 Jun 2021, at 2:13 PM, Reej M  wrote:
> 
> Oh ok Walter. 
> For the moment, we too cannot update to cloudsolrclient, and we are trying to 
> find a way to resume the connections for now, and later work on the code 
> cleanup. Thanks
> 
>> On 30 Jun 2021, at 12:49 AM, Walter Underwood  wrote:
>> 
>> CloudSolrClient is not an absolute requirement for a Solr Cloud cluster. 
>> 
>> We use regular HTTPSolrClient sending all requests to the load balancer.
>> Actually, we use a separate load balancer for indexing, to keep the 
>> monitoring
>> separate and to set different timeouts than for queries.
>> 
>> This setup is simple and fast. With our biggest cluster, we index about a 
>> half million documents per minute.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>>> On Jun 29, 2021, at 9:37 AM, Reej Nayagam  wrote:
>>> 
>>> Thanks Shawn & Vicenzo. Will check it out and change accordingly. Thanks
>>> again Shawn for your clear explanation.
>>> 
>>> Regards
>>> Reej
>>> 
>>> 
>>> On Tue, 29 Jun 2021 at 9:47 PM, Vincenzo D'Amore  wrote:
>>> 
>>>> Right, you should always use CloudSolrClient as a singleton.
>>>> To be honest I'm used to reuse a CloudSolrClient instance for each
>>>> collection/index.
>>>> 
>>>> On Tue, Jun 29, 2021 at 3:12 PM Shawn Heisey  wrote:
>>>> 
>>>>> On 6/29/2021 6:43 AM, Reej Nayagam wrote:
>>>>>> Hi Vincenzo Yes we are using cloud and initial solr version was 4.10.4
>>>>>> and we upgraded the jars alone to 8.8.2 now in the application side
>>>>>> connecting to solr Server to fix some vulnerability. As we have
>>>>>> upgraded the jars we changed httpsolrserver connection to
>>>>>> httpsolrclient and we guess there is some connection leak and wanted
>>>>>> to check if we need to close it or it is being handled internally.
>>>>>> Singleton not sure if I can use as the base URL changes depending on
>>>>>> the leader.
>>>>> 
>>>>> If you're running SolrCloud, you should be using CloudSolrClient, not
>>>>> HttpSolrClient.  The cloud client talks to zookeeper, so it is always
>>>>> aware of the cluster state -- it will be aware of down servers and new
>>>>> servers, without recreating or reinitializing the client.  And it will
>>>>> be aware of changes instantly -- because that information is coordinated
>>>>> in zookeeper.
>>>>> 
>>>>> You can use a single client object for multiple collections.  All of the
>>>>> methods that execute requests should have a version where you can pass
>>>>> it the name of the collection/core you want to operate on. For
>>>>> CloudSolrClient, you point it at all your ZK servers and it figures out
>>>>> the Solr server URLs  from the clusterstate in ZK.  For HttpSolrClient,
>>>>> you just leave the collection name off of the base URL --
>>>>> "http://server.example.com:8983/solr"; is an example URL.
>>>>> 
>>>>> As was mentioned, you should create a client during program startup and
>>>>> then use it to handle all requests for the life of the program. It
>>>>> should manage connections and close them after receiving data, with no
>>>>> coding needed from the developer (you).  If you close a SolrClient
>>>>> object, it will not function after that.  If you're having connections
>>>>> stay open, then either you're running a Solr or SolrJ version with bugs,
>>>>> or there is something wrong with your networking.
>>>>> 
>>>>> It shouldn't be necessary to ever close a SolrClient object, unless you
>>>>> create a new one every time your program talks to Solr.   Which you
>>>>> shouldn't do.
>>>>> 
>>>>> 
>>>>> Thanks,
>>>>> Shawn
>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> Vincenzo D'Amore
>>>> 
>>> -- 
>>> *Thanks,*
>>> *Reej*
>> 
> 

Could not load collection from ZK (worked in solr4, failing in solr8) - Help Needed

[Reej M]

Hi All,

We are upgrading from solr 4 to solr 8. Updated the solr server to solr8.8.2, 
solrj8.8.2.
We are trying to resume the code and in one place where we are trying to get a 
collection from clusterstate. Code used is as below

Clusterstate clusterState = null;
For(SearchSolrZookeeper zk:livezks){
SolrZkClient zkClient = null;
ZKStateReader stateReader = null;
Try{
Zkclient = new SolrZkClient(zk.getUrl(), 3000);
stateReader = new ZKStateReader(Zkclient);
stateReader.createClusterStateWatchersAndUpdate();
clusterState = stateReader.getClusterState();
Connected = true;
Exception = null;
break;
}
Catch(){
}
Finally{
State reader.close();
Zkclient.close();
}

If(clusterstate!=null){
Doccollection dc = clusterstate.getCollection(“INVOICE”);
//here dc is null
}
}


Tried using getcollectionornull - returns null
Getcollectionornull(collection name, true) — returns null

When I debug I can see clusterstate has below values in it

Znodeversion:0
Live node:[192.123.33.44:3883_Solr, 
192.123.33.44:9883_Solr,192.123.33.45:3883_Solr,192.123.33.45:3883_Solr]
Collections:{ORDER=LazyCollectionRef(ORDER), INVOICE= 
LazyCollectionRef(INVOICE), PAYMENT= LazyCollectionRef(PAYMENT),etc..}

From the document collection we get the slices and iterate each replica to show 
the cores in a admin page. But this page worked fine with the code in solr4, 
now with solr8 it throws error. Kindly help. 

Regards
Reej

Re: Could not load collection from ZK (worked in solr4, failing in solr8) - Help Needed

[Reej M]

> On 8 Jul 2021, at 7:00 PM, Reej M  wrote:
> 
> Hi All,
> 
> We are upgrading from solr 4 to solr 8. Updated the solr server to solr8.8.2, 
> solrj8.8.2.
> We are trying to resume the code and in one place where we are trying to get 
> a collection from clusterstate. Code used is as below
> 
> Clusterstate clusterState = null;
> For(SearchSolrZookeeper zk:livezks){
> SolrZkClient zkClient = null;
> ZKStateReader stateReader = null;
> Try{
>Zkclient = new SolrZkClient(zk.getUrl(), 3000);
>stateReader = new ZKStateReader(Zkclient);
>stateReader.createClusterStateWatchersAndUpdate();
>clusterState = stateReader.getClusterState();
>Connected = true;
>Exception = null;
>break;
> }
> Catch(){
> }
> Finally{
> State reader.close();
> Zkclient.close();
> }
> 
> If(clusterstate!=null){
> Doccollection dc = clusterstate.getCollection(“INVOICE”);
> //here dc is null
> }
> }
> 
> 
> Tried using getcollectionornull - returns null
> Getcollectionornull(collection name, true) — returns null
> 
> When I debug I can see clusterstate has below values in it
> 
Using SOLR8

> Znodeversion:0
> Live node:[192.123.33.44:3883_Solr, 
> 192.123.33.44:9883_Solr,192.123.33.45:3883_Solr,192.123.33.45:3883_Solr]
> Collections:{ORDER=LazyCollectionRef(ORDER), INVOICE= 
> LazyCollectionRef(INVOICE), PAYMENT= LazyCollectionRef(PAYMENT),etc..}

When Debugging with Solr4 clusterstate is as below
> Znodeversion:134
> Live node:[192.123.33.44:3883_Solr, 
> 192.123.33.44:9883_Solr,192.123.33.45:3883_Solr,192.123.33.45:3883_Solr]
> Collections:{ORDER(ORDER/clusterstate.json/134)={
“Replication factor”:”1”,
“Shards”:{“shard1”:{
Range:
“State”:”active”,
“Replicas”:{
“Core_node1”:{
“Core”:ORDER_shard1_replica1
“Base_url:
“Node_name”:”http://_solr <http://_solr/>”
“State”:”active”
“Leader”:”true”},
“Core_node2”:{
“Core”:ORDER_shard1_replica1
“Base_url:
“Node_name”:”http://_solr <http://_solr/>”
“State”:”active”
“Leader”:”true”}
}
}
> 
> From the document collection we get the slices and iterate each replica to 
> show the cores in a admin page. But this page worked fine with the code in 
> solr4, now with solr8 it throws error.




> Kindly help. 
> 
> Regards
> Reej

MaxConnections and MaxconnectionsPerHost. (Solr 4 and Solr 8)

[Reej M]

Hi Team,

Want an advise on this property maxConnections and maxconnectionsperhost.
We were  using solr 4 and on searching in google I found the default max 
connections per host is 20 in solr4 , and it is 10in solr 8. Can you please 
help me to understand the below
1. So in Solr 4, only 20 requests per hosts are allowed at a time (document 
does not have a mention about this)
2. If Solr 8 allows 10 connections per host and will that not slow down the 
system.
maximumPoolSize - The maximum size of the threadpool servicing requests. 
Default is unlimited - So if the requesting application is WebLogic, so it 
depends on the WebLogic thread pool size?

Kindly advise if we need to limit all these values in solr8 so it does not slow 
down the system, because we are moving to solr 8 and the WebLogic Server is 
going down within 30 minutes and we dont see any errors in WebLogic side or 
solr side, so we think the connections are the issue , but solr 4 we didn’t 
have any issues with the connection

Thanks
Reej

Cloudsolrclient.getclusterstateprovider - returns incorrect base_url [http instead of https] - Urgent pls help

[Reej M]

Hi Team,

We have enabled ssl. Solr version 8.8.2, zk - 3.6.3
When we try to get the base url using cloudsolrclient.getclusterstateprovider, 
it returns http url instead of HTTPS.
Please find the screenshots while I debug,

Code used


From solr admin server - using 
/solr/admin/collection?action=CLUSTERSTATUS&wt=json return https as base url


Kindly help to fix this. Thanks

Re: Cloudsolrclient.getclusterstateprovider - returns incorrect base_url [http instead of https] - Urgent pls help

[Reej M]

> On 21 Jul 2021, at 5:07 PM, Vincenzo D'Amore  wrote:
> Hi,
Is ok sometime all of us just loose our cool.
By the way we have followed the same steps as per the documentation only. 
Im trying to clear the zk data, clear everything and recheck again if that 
might help. Thanks
 
> this is your version,
> https://solr.apache.org/guide/8_8/enabling-ssl.html#EnablingSSL-SolrCloud
> anyway, pay attention to clusterprop
> https://solr.apache.org/guide/8_8/enabling-ssl.html#update-cluster-properties-for-existing-collections
> 
> On Wed, Jul 21, 2021 at 11:04 AM Vincenzo D'Amore 
> wrote:
> 
>> Have you double checked how ssl has been configured?
>> I think this doc could help
>> https://solr.apache.org/guide/6_6/enabling-ssl.html#EnablingSSL-SolrCloud
>> 
>> And double check the "clusterprop" urlScheme
>> 
>> On Wed, Jul 21, 2021 at 11:01 AM Vincenzo D'Amore 
>> wrote:
>> 
>>> Sorry for being so rude, I was in a hurry and just a while after the mail
>>> was sent I thought you weren't in the position to take a screenshot.
>>> 
>>> 
>>> On Wed, Jul 21, 2021 at 10:41 AM Reej Nayagam  wrote:
>>> 
>>>> Hi Sorry,
>>>> My offic system doesn't have internet for security reasons.
>>>> The issue is we have enabled ssl in solr and we try to get the
>>>> clusterstate
>>>> as  below
>>>> cloudsolrclient.getclusterstateprovider.getclusterstate()
>>>> then using the clusterstate doccollection
>>>> DocCollection dc = clusterstate.getCollection(collectionName)
>>>> for(Slice slice:dc.getSlices()){
>>>>  for(Replica replica:slice.getReplicas()){
>>>> replica.get("base_url")
>>>> }
>>>> }
>>>> 
>>>> replica.get("base_url") -- returns a http url but what we have configured
>>>> is a https url
>>>> solr admin page we are able to access using https url and can see the
>>>> clusterstatus with base_url having https node url
>>>> Not sure how through java it is different
>>>> 
>>>> 
>>>> *Thanks,*
>>>> *Reej*
>>>> 
>>>> 
>>>> On Wed, Jul 21, 2021 at 4:19 PM Vincenzo D'Amore 
>>>> wrote:
>>>> 
>>>>> your screenshots are unreadable, couldn't you copy and paste the text
>>>> from
>>>>> the screen?
>>>>> 
>>>>> On Wed, Jul 21, 2021 at 9:18 AM Reej M  wrote:
>>>>> 
>>>>>> Hi Team,
>>>>>> 
>>>>>> We have enabled ssl. Solr version 8.8.2, zk - 3.6.3
>>>>>> When we try to get the base url using
>>>>>> cloudsolrclient.getclusterstateprovider, it returns http url instead
>>>> of
>>>>>> HTTPS.
>>>>>> Please find the screenshots while I debug,
>>>>>> Code used
>>>>>> 
>>>>>> From solr admin server - using
>>>>>> /solr/admin/collection?action=CLUSTERSTATUS&wt=json return https as
>>>> base url
>>>>>> 
>>>>>> Kindly help to fix this. Thanks
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Vincenzo D'Amore
>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> Vincenzo D'Amore
>>> 
>>> 
>> 
>> --
>> Vincenzo D'Amore
>> 
>> 
> 
> -- 
> Vincenzo D'Amore

Re: ConcurrentUpdateSolrClient stall prevention bug in Solr 8.4+

[Reej M]

Hi Team,

Have any of you  found a solution for the Task Queue  processing has stalled 
for 20077 ms with 0 remaining elements to process.
We are using solr 8.8.2, randomly we get this error while indexing. Is there 
any way we need to tune the solr.autocommit.maxtime?
For few cores we have it as 15000, for few we have it as 500, 1 or -1.  Also 
the maxwarmingsearchers is set to 2
Kindly advise on how to tune this. 

Thanks
Reej

Re: Help Needed: Distributed update Async Exception solr 8.8.2 - Update

[Reej M]

Hi All,

I’ve went through the  section "Ignoring Commits from Client Applications in 
SolrCloud” in the solr 8.8 documentation.
Our  Java application is issuing a commit after indexing, so as of I understood 
from the document, when running in cloud mode, the applications should not 
explicitly send a commit request.
So as mentioned in the doc, I’ve updated the solrconfig.xml with the  
updateRequestProcessorChain with the class 
IgnoreCommitOptimizeUpdateProcessorFactory.
After doing this, I’ve not seen the Async exception during distributed update 
error yet for the past 30 minutes.

So my doubt now is, Do we need to do any changes or wire anything for the 
optimisation and commit to be carried out effectively, as we are ignoring 
commit here or does solr internally will handle it.
Our auto commit maxtime is 6 and Opensearcher - false
Auto soft commit maxtime : 18 .

Please let us know if any changes are required in the config.

Thanks
Reej





> On 22 Aug 2021, at 7:10 PM, Reej Nayagam  wrote:
> 
> Hi All,
> We are getting a distributed update async exception during indexing. Solr 
> version v8.8.2 and we are using cloudsolrclient and passing 3 zk ip’s for 
> connecting. This indexing happens every 15 mins and at times the entire 
> indexing is getting wiped out with data being pulled from the DB.
> 
> Earlier we are using v4.10 and for indexing we used find the leader URL and 
> connect for indexing and we didn’t have any issues with it. 
> 
> We need to push this new solr version 8.8.2 to prod and this issue is holding 
> us back
> 
> Please advise or help if there is any workaround.
> 
> 
> Full Import 
> failed:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>  Async exception during distributed update: Error from server at 
> https://192.168.3.153:3883/solr/PUBLIC_shard1_replica_n1/ 
> : Server Error
> request: https://192.168.3.153:3883/solr/PUBLIC_shard1_replica_n1/ 
> 
> Remote error message: Task queue processing has stalled for 20217 ms with 0 
> remaining elements to process.
> at 
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1193)
> at 
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1102)
> at org.apache.solr.handler.dataimport.SolrWriter.close(SolrWriter.java:61)
> at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:282)
> at 
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:427)
> at 
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:486)
> at 
> org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:469)
> at java.lang.Thread.run(Thread.java:748)
> 
> -- 
> Thanks,
> Reej

Solr Windows Service / Running solr in the background

[Reej M]

Hi All,

For now we are running solr cloud starting it from the command prompt as below 
from the 2 servers for starting the 2 nodes

Server 1 & Server 2
bin\solr start -cloud -p 3883 -s “server\solr\cloud\node\solr”
bin\solr start -cloud -p 9883 -s “server\solr\cloud\node\solr”

But if we close the command prompt server will obviously go down, is there any 
way we can have this running in the background.

We tried using shell script, but we can run it as a standalone service, but is 
not working when we pass the commands to start as a cloud service.
We followed  http://lets-share.senktas.net/2017/11/solr-as-a-service.html 
 for running as 
windows service.

Any one has tried any options for starting solr cloud as windows service, do 
share your thoughts.

Thanks
Reej

No live solr servers & Invalid key request timestamp error

[Reej M]

Hi All,

We are using solr cloud (V8.8.2). Yesterday we noticed this error from 3 am to 
4am
PKIAuthenticationPlugin : Invalid Keyrequest timestamp:1638906938935, received 
timestamp : 1638906962821 TTL:15000

Also, during indexing, we are having the exception  “No live SolrServers 
available to handle this request : https://x.x.x.x:3993  
and https://x.x.x.x:9983 
But when we check through the solr cloud admin UI, the nodes are all green and 
active.

Exception occurs when we call this line — QueryREsponse response = 
client.query(params).

Client is SolrClient and params values are set as below
ModifiableSolrParams params = new ModifiableSolrParams();
params.set(“qt”,corename);
params.set(“command”,”INDEX”);

Exception caused by org.apache.solr.client.solrj.solrserverexception: No live 
SolrServers available to handle this request : https://x.x.x.x:3993 
 and https://x.x.x.x:9983 
At org.apache.solr.client.solrj.impl.LBSolrclient.request(LBSolrClient.java:346)
..
..
..
..
At org.apache.solr.client.solrj.solrclient.query(solrclient.jav:1018)

Much appreciate your advise. Please note haven’t encountered this error anytime.

Thanks
Reej