Solr High Memory and CPU Usage

Hi everyone, We are using solr 8.1.1. We see memory and CPU are highly utilized (~ 90%) for all the solr nodes every day. We have solr cluster of 8 servers. Each server has 2 instances running. Please let me know how to proceed and what actions to take. Let me know if more information is needed.

Re: Solr High Memory and CPU Usage

____ > > > > De: HariBabu kuruva > Enviado: viernes, 23 de abril de 2021 8:39 > Para: users@solr.apache.org > Asunto: Solr High Memory and CPU Usage > > Hi everyone, > > We are using solr 8.1.1. We see memory and CPU

query multiple collections at a time

Hi Everyone, We generally query a solr store (collection) in the solr UI by selecting each collection. I would like to query multiple collections at a time, do we have any backend command line scripts/commands. Could you please share it really helps. Please let me know if more information is req

Re: query multiple collections at a time

gt; http://yourSolrAddress:8983/solr/collectionA,collectionB,collectionC/select > > hope it helps > > -ufuk yilmaz > > Sent from Mail for Windows 10 > > From: HariBabu kuruva > Sent: 23 April 2021 15:38 > To: users@solr.apache.org > Subject: query multiple collectio

Solr heap memory

Hi All We are using solr cloud 8.8.1. We are frequently seeing memory and CPU reaching 100%. Can someone please suggest ideal way to define java heap memory. Server memory is 62Gb, swap-7Gb Currently we have defined as below in solr in.sh SOLR_HEAP= 46000m SOLR_JAVA_MEMORY= Xms:1024m, Xmx: 70

Re: Solr heap memory

now if it helps > > > On Aug 29, 2021, at 3:56 AM, HariBabu kuruva > wrote: > > > > Hi All > > > > We are using solr cloud 8.8.1. > > > > We are frequently seeing memory and CPU reaching 100%. > > > > Can someone please suggest id

Re: Solr heap memory

stions/3661/why-is-a-32-gb-heap-allocation-not-recommended.html > > But this is almost like a protected secret > > > On Aug 29, 2021, at 1:52 PM, Shawn Heisey wrote: > > > > On 8/29/2021 2:38 AM, HariBabu kuruva wrote: > >> Is it required to define both the para

Re: Solr heap memory

to standalone solr instance. > > > > > On Aug 30, 2021, at 3:23 AM, HariBabu kuruva < > hari2708.kur...@gmail.com> wrote: > > > > > > Hi Dave > > > > > > We tried setting the memory as per your suggestions. > > > > > &g

Re: Solr heap memory

HariBabu kuruva wrote: > Hi Zisis, > > Thanks for your email. > > We are suspecting the issue with one particular solr collection(or > store). Wherever the replicas of that store are present that nodes are > going down. > > Also now that shard is in recovery mode and L

Solr 8.8.1 | Duplicate values

Hi All, We have recently upgraded solr cloud from 8.1 to 8.8.1. After upgrade we are facing duplicate values in solr data. Please find the below screenshot for your reference. Could someone please suggest ways to avoid the issue. [image: image.png] -- Thanks and Regards, Hari Mobile:97907565

Solr | OOM | Lazyfield

Hi All, We are getting OOM errors in the solr logs for only specific solr stores. And in the solr logs we see the below error. Is the OOM error could be because of the below error. Also i see below Lazyfield error is spanned across thousands of lines. Please advise. This is a PROD environment.

Re: Solr | OOM | Lazyfield

dexing - attempting to add multiple values to a field > that hasn't been defined in the schema as multivalued. I'd review your > indexing process and schema first. > > Charlie > > On 07/09/2021 11:00, HariBabu kuruva wrote: > > Hi All, > > > > We are g

Re: Solr | OOM | Lazyfield

nt to the field 'quoteSLOCurrentStatus' when > it's probably not defined as multivalued. > > C > > On 07/09/2021 12:30, HariBabu kuruva wrote: > > Thank you Charlie. > > > > Please let me know if anything is required from my end. > > > > A

Re: Solr | OOM | Lazyfield

Charlie Hull wrote: > I don't know, but perhaps 8.8.1 checks it more than 8.1? - it's still > worth your dev team reviewing as I suggested as it may be linked to why > the nodes are going down. The error is occurring for a reason. > > Charlie > > On 07/09/2021 14:30

Solr High Availability | Multi-Datacenter approach

Hi All, We have Solr cloud running on 10 nodes with Zookeeper running on 5 nodes. They are running across 2 data centers. As part of high availability we would like to run solr so that it should sustain even if one data center goes down. How can we achieve this approach as we need to maintain th

Solr Error

Hi All, Am getting below error in the solr , which is affecting the services which are accessing the dqmStore. Please suggest if you have come across this issue. 2022-03-24 08:48:36.629 ERROR (qtp1198197478-620853) [c:dqmStore s:shard1 r:core_node4 x:dqmStore_shard1_replica_n2] o.a.s. u.SolrCmd

Solr dynamic reconfiguration of zookeeper ensemble

Hi All, We are using solr 8.11.1 with zk 3.2. Recently we have upgraded zk to 3.7.1. We observe below warning on solr-ui . "Solr ZK connection string (2 hosts) is different from the dynamic ensemble config (3 hosts)." Please let us know will it cause any issue and Is there any option to make so

Re: Solr dynamic reconfiguration of zookeeper ensemble

Hi All, Can someone please help with the above query. Also after the ZK upgrade, when i see the zk status in solr-ui, it is throwing the error : java.net.SocketException: Connection reset Could you please help me. On Mon, Jul 11, 2022 at 8:28 PM HariBabu kuruva wrote: > Hi All, > &g

Re: Solr dynamic reconfiguration of zookeeper ensemble

Hi All, Anyone please let me know if you have come across the below issue. Please help me if you have any ideas on this. On Sun, Jul 24, 2022 at 10:52 AM HariBabu kuruva wrote: > Hi All, > > Can someone please help with the above query. > > Also after the ZK upgrade, when i se

Re: Solr dynamic reconfiguration of zookeeper ensemble

armless as it is only affecting > the UI, but it will be solved by upgrading to Solr 8.11.2, see > https://issues.apache.org/jira/browse/SOLR-15849 > > Jan > > > 11. jul. 2022 kl. 16:58 skrev HariBabu kuruva >: > > > > Hi All, > > > > We are using so

Solr Upgrade

Hi All, I would like to upgrade Solr from 8.11.1 to 8.11.2. Please help me with the steps I should follow so that the existing solr data is not affected. Thanks. -- Thanks and Regards, Hari Mobile:9790756568

Re: Solr Upgrade

r change so I would suggest you to ignore this upgrade. > > On Thu, 11 Aug 2022 at 11:26 AM, HariBabu kuruva < > hari2708.kur...@gmail.com> > wrote: > > > Hi All, > > > > I would like to upgrade Solr from 8.11.1 to 8.11.2. > > > > Please help me wi

Re: Solr dynamic reconfiguration of zookeeper ensemble

Hi Jan, Is it wise to upgrade solr in PROD to 8.11.2 from 8.11.1 only for that zookeeper error ? What are your thoughts? On Fri, Aug 5, 2022 at 6:45 PM HariBabu kuruva wrote: > Thank you very much for the reply. > > On Fri, Aug 5, 2022 at 5:43 PM Jan Høydahl wrote: > >> H

Re: CVE-2022-42889

Hi All, We are using solr-8.11.1 , Currently we have commons-text-1.6.jar, shall i replace it with commons-text-1.10.0.jar and restart the application? Will that work ? Please help. On Wed, Oct 19, 2022 at 3:28 PM Markus Jelsma wrote: > Yes, it is already being done: > https://issues.apache.o

PROD Solr errors

Hi All, We are receiving below errors when the manual indexing is initialized for one of the stores, and its working fine for all other stores. The certs are updated and in place. *Architecture*: 10 solr nodes and 5 zookeepers. 60 GB Ram and 4 cpu for all 10 nodes. ~48 GB Heap is assigned. *

Re: PROD Solr errors

your data. The broken pipe > messages indicate the same, it happens when connections are broken. > > Find out which servers went offline/unavailable, and see what they wrote in > their logs. > > Regards, > Markus > > Op di 14 feb. 2023 om 13:00 schreef HariBabu kuruva <

Re: PROD Solr errors

ou into that setting nevermind. Certainly > sometimes super-sized heap is necessary, but usually not. If that setting > was chosen via "hmmm, this looks good" ... it doesn't look good to me ;) > > On Thu, Feb 16, 2023 at 6:21 AM HariBabu kuruva > > wrote: > > >

Re: PROD Solr errors

.4.44.v20210927] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.44.v20210927.jar:9.4.44.v 20210927] On Fri, Feb 17, 2023 at 9:27 AM HariBabu kuruva wrote: > Hi > > Thank you for your reply. I will try to check if there was any network

PKIAuthenticationPlugin error

I see the below error repeatedly in our Solr logs. Please suggest how to avoid this. ERROR true PKIAuthenticationPlugin Invalid key request timestamp: 1677401295538 ,​ received timestamp: 1677401316679 ,​ TTL: 15000 -- -- Thanks and Regards, Hari Mobile:9790756568

Solr Data Import | inconsistent output

Hi All, Our Developers have initiated data import. And it shows different outputs on each refresh. One time it shows as Data import is completed and the other time it says in progress. Screenshots below. Please help. [image: image.png] [image: image.png] -- Thanks and Regards, Hari Mobile:9

Re: PKIAuthenticationPlugin error

> > Thomas > > Op zo 26 feb. 2023 12:24 schreef HariBabu kuruva < > hari2708.kur...@gmail.com > >: > > > I see the below error repeatedly in our Solr logs. Please suggest how to > > avoid this. > > > > > > ERROR true PKIAuthenticationPlu

Solr Heap Memory Settings

Hi All, I have set the Heap memory as -Xms 1g -Xmx 40g in the Production environment. But when i see the Heap memory in the Solr UI. I can see the Max Heap below. Max: 3.8Gb Used: 2.2Gb Could you please help me understand, how is it showing the Max Heap as 3.8Gb. I have 10 solr nodes, it shows

Re: Solr Heap Memory Settings

gt;> > >>>> Deepak > >>>> "The greatness of a nation can be judged by the way its animals are > treated > >>>> - Mahatma Gandhi" > >>>> > >>>> +91 73500 12833 > >>>> deic...@gmail.com > >&

Re: Solr Heap Memory Settings

choice by some observations or measurements? > > Jan > > > 14. mar. 2023 kl. 06:39 skrev HariBabu kuruva >: > > > > Thank you all for your responses. > > > > There are no spaces between Xms and the values. > > > > I have updated similar

Solr manual indexing errors

Hi All, Our developers are doing manual indexing. And we see the below errors in the logs. Could you please help with what is causing the issue. *IOException while taking to the to server at : https:hostname/solr/Storename* *Caused by: javax.net..ssl.SSLPeerUnverifiedException: peer not authentic

Re: Solr manual indexing errors

d.java:834) WARN vmc** 14-03-2023 09:54:50.145 [mainRequestHandler-1] eisp.kafka.cdc.task.MainIndexerTask.handleBulkDataRequest - Bulk Request Taken Care Completely! On Tue, Mar 14, 2023 at 7:59 PM Shawn Heisey wrote: > On 3/14/23 08:09, HariBabu kuruva wrote: > > Our develo

Solr Shard not coming up

Hi All, I have come across an issue where a few shards are not coming up after solr restart. Currently I don't have an error message stored. But I remember it was due to some index file corruption. Please share your suggestions to avoid such issues. -- Thanks and Regards, Hari Mobile:9790756

Re: Solr Shard not coming up

corruption is very rare, so mys supect is unclean shutdown, > antivirus quarantine, locking issues etc. Please provide more details. > > Jan > > > 23. mar. 2023 kl. 07:54 skrev HariBabu kuruva >: > > > > Hi All, > > > > I have come across an issue wher

Solr Node High CPU

Hi All, We have 4 solr nodes, only one of the nodes is consuming almost High CPU from yesterday. We don't see any errors in the logs. Restart dint resolve the issue. Could you please help me to identify the issue. Solr Version: 8.11.1 Instances: Single instance per machine. No other softwares run

Re: Solr Node High CPU

It's the solr process consuming the 95% CPU. I want to know how to troubleshoot this from the solr end. On Fri, Mar 24, 2023 at 5:03 PM Deepak Goel wrote: > can you check which process is eating up the CPU using top command > > On Fri, 24 Mar 2023, 15:54 HariBabu kuruva, > wr

Re: Solr Node High CPU

ation on the server (high cpu) has gone > > astray. You might want to reinstall it (including updating all > libraries). > > > >> On Fri, 24 Mar 2023, 18:31 Mikhail Khludnev, wrote: > >> > >> SolrAdmin Thread Dump, or jstack might give a clue. > >>

Query on Solr and ZK ports

Hi All, I could see the solr process is Listening on 7981 port along with the normal solr port(8981). It is shown as DSTOP PORT , when I grep solr process. Could you please give more details on this port, Can we disable this ? With regards to Zookeeper I could see port 8080 as a ZK admin port, Ho

Re: Query on Solr and ZK ports

Can someone please help with this information? On Tue, Apr 4, 2023 at 10:01 PM HariBabu kuruva wrote: > Hi All, > > I could see the solr process is Listening on 7981 port along with the > normal solr port(8981). It is shown as DSTOP PORT , when I grep solr > process. Could you p

Re: Query on Solr and ZK ports

and ZK, but > > between the Zookeepers, also ports 2888 and 3888 needs to be open for > > internal communication, see > > > https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.0/bk_reference/content/zookeeper-ports.html > > I'm not aware of other ports. > > > >

Unable to create collections

Hi All, I am unable to see any of the Solr host stats(like CPU/Mem..) in the UI, though the zookeepers and Solr services are running fine. *Cluster Details:* 2 solr Nodes: (running on different hosts) 3 Zookeepers: (running on different hosts) CPU: 4 for all nodes Ram: 60 GB for solr host1 and 30

Re: Unable to create collections

> Hi, > > Have you configured SSL on the Solr hosts? If so, you need to set the > urlScheme cluster property accordingly. > > Jan > > > 12. mai 2023 kl. 06:23 skrev HariBabu kuruva >: > > > > Hi All, > > > > I am unable to see any of th

Solr Error| cluster state says we are the leader but locally we don't think so

Hi All, As part of the O.S patching we have rebooted the servers and services in PROD environment. After the activity we have started our services and we see below errors in Solr. Remote error message: ClusterState says we are the leader (https://solrhostname.corp.equinix.com:port/solr/abcStore_sh

Add a new Shard to the collection

Hi All, I would like to add a new shard to the existing collection to have better performance. Currently we have one shard. Solr - 8.11.1 Nodes(servers) - 10 (Non prod - 4 nodes) Zookeepers-5 I have tried the SPLITSHARD command in one of the non prod environments. *https://solrserver.corp.comp

Re: Add a new Shard to the collection

Hi All, Could someone help with this question? On Mon, Jul 24, 2023 at 8:44 PM HariBabu kuruva wrote: > Hi All, > > I would like to add a new shard to the existing collection to have better > performance. Currently we have one shard. > > Solr - 8.11.1 > Nodes(server

Re: Add a new Shard to the collection

s well it's ok to have old > shard inactive. > You can request DELETESHARD to reclaim the disk space. > > On Mon, Jul 24, 2023 at 6:19 PM HariBabu kuruva > > wrote: > > > Hi All, > > > > I would like to add a new shard to the existing collection to have be

Re: Add a new Shard to the collection

++ FYI, I can see the old shard automatically removed. On Mon, Jul 31, 2023 at 11:39 AM HariBabu kuruva wrote: > Thanks for your reply. > > I am a little bit worried about PROD. Can I go ahead and do the same steps > in PROD ? Do I need to take any backups or any steps before doing

Re: Add a new Shard to the collection

Hi All, I did sharding, splitted shard1 into shard-1_0 and shard-1_1 I want to have one more shard(3 shards). In this case, which shard should I split . Please advise. On Tue, Aug 1, 2023 at 11:17 AM HariBabu kuruva wrote: > ++ FYI, I can see the old shard automatically removed. > &g

Re: Add a new Shard to the collection

of shards? > > What has improved as you moved from one to two shards? Why decide then that > you then want to have 3 shards and no stay at 2 or move to 4? > > Ilan > > On Wed, Aug 2, 2023, 5:48 PM HariBabu kuruva > wrote: > > > Hi All, > > > > I did

Re: Add a new Shard to the collection

> If your nodes are saturated by traffic, you need more nodes (and more > replicas so that the added nodes have a replica as well). > > Ilan > > On Thu, Aug 3, 2023, 8:23 AM HariBabu kuruva > wrote: > > > Hi Ilan, > > > > Thank you for your reply. > >

Re: Add a new Shard to the collection

ou need more nodes (and more > replicas so that the added nodes have a replica as well). > > Ilan > > On Thu, Aug 3, 2023, 8:23 AM HariBabu kuruva > wrote: > > > Hi Ilan, > > > > Thank you for your reply. > > > > Application requests are f

Request processing has stalled

Hi All, We are facing the below error in our PROD environment. Could you please let me know what are all the causes for this issue? and how to troubleshoot it. We have 10 solr nodes. Heap Memory: 30GB Total VM Memory:60 GB Solr Version: 8.11.1 *CmdDistributor java.io.IOException: Request proces

Idle timeout expired

Hi All, We are seeing below errors in our solr logs. Could you please suggest me what are all the causes for this issue and how to deal with it? *ERROR:* Caused by: java.util.concurrent.*TimeoutException: Idle timeout expired*: 12/12 ms at org.eclipse.jetty.io.IdleTimeout.checkIdl

Re: About Solr 9.1.1 ERROR "o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling SolrCmdDistributor$Req: xxx => java.io.IOException: java.util.concurrent.TimeoutException: Total timeout

Hi All, Am facing the same error in the PROD environment. It would be very helpful if someone helps to figure out the issue. On Tue, May 16, 2023 at 8:45 AM Mingchun Zhao wrote: > Hello, > > We have recently seen the below errors repeated when update requests were > received by Solr. Could you

Solr-9.5 | collection creation error

HariBabu kuruva 11:07 AM (8 minutes ago) to users.solr Hi All, I am working on a POC, to set up solr-9.5.0. Currently I have Solr-8.11.1 installed and running. As part of the POC. I have installed Solr-9.5.0(on one node) and Zookeeper 3.9.2 (on 3 nodes). This setup is done on the same server

Solr-9.5 | collection creation error

HI , I am working on a POC, to set up solr-9.5.0. Currently I have Solr-8.11.1 installed and running. As part of the POC. I have installed Solr-9.5.0(on one node) and Zookeeper 3.9.2 (on 3 nodes). This setup is done on the same server where I have solr-8.5 running but with different ports. Solr

Re: Solr-9.5 | collection creation error

t configset from the UI? > - Did the Solr node crash or recover during this process? > - Are there ERRORs in Zookeeper's log files? > > Jan > > > 17. apr. 2024 kl. 02:37 skrev HariBabu kuruva >: > > > > HI , I am working on a POC, to set up solr-9.5.0. C

Re: Solr 9.1.1 - Task queue processing

Hi, Can you please try increasing stalltime by adding the below parameter in the startup script. *-Dsolr.cloud.client.stallTime=6* Also try to analyze the indexing load on the solr nodes and see if you can reduce it. On Wed, Apr 17, 2024 at 1:58 PM Stoyanov, Nikolay wrote: > Hello, > I am

Re: Solr-9.5 | collection creation error

Hi All, Could you please help on the above issue. On Wed, Apr 17, 2024 at 2:39 PM HariBabu kuruva wrote: > Hi Jan, > > Thanks for the response. > > Please find the details below: > > *Exactly how did you install Solr, and what is your ZK_HOST connection > string?:

Re: Solr-9.5 | collection creation error

1119242936-20-solr_hostname.corp.abc.com-123) [c:workdayUserStore s: r: x: t:solr_hostname.corp.abc.com-123] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&maxShardsPerNode=1&collection.configName=workdayUserStoreConfig&name=workdayUserStore&action=CREATE&

Re: Solr-9.5 | collection creation error

gt; 4. Install Zookeeper as follows > 5. Start Solr with systemctl start solr > 6. etc. > > Jan > > > 17. apr. 2024 kl. 11:09 skrev HariBabu kuruva >: > > > > Hi Jan, > > > > Thanks for the response. > > > > Please find the details