Re: Requesting help from the community on GC config testing

Hi Shawn, I can try to help you with the test. I have a 6 solr node cluster ( machines with 4 cores and 28GB RAM, 250 GB hard disk ) running on OpenJDK 11.0.11) having 2 shards and 3 replica's each. Currently, the cluster has 27GB of data per core, I can ingest more data to make it around 100GB pe

Re: Solr 8 autocommit vs client commits

Hi Saur, Yes , if you commit from a client with time set as 30 sec then the pending docs will also get committed in solr. Sending explicit commit requests from client applications should be avoided as it can severely impact the performance of your clusters. Solr provides IgnoreCommitOptimizeUpdat

Concurrent transactions and autocommit in Solr 8

Hi, I have a (noobie) question about Solr 8 autocommit behaviour. This is my scenario: - Autocommit configured in solrconfig - ETL job that indexes thousands of documents whenever it runs The ETL job updates the collection in the following manner: first it deletes a series of documents based o

Re: Solr 8 autocommit vs client commits

Many thanks for the clarification! From: dinesh naik Sent: 13 October 2021 10:14 To: users@solr.apache.org Subject: Re: Solr 8 autocommit vs client commits *** External email: use caution *** Hi Saur, Yes , if you commit from a client with time set as 30 sec

Re: Concurrent transactions and autocommit in Solr 8

Hello If the insert/delete are done with two different threads (and with no synchronization), it could be possible that the new records are getting deleted. We might have to dig a bit into Solr code I guess. Deepak "The greatness of a nation can be judged by the way its animals are treated - Maha

Re: Solr keeps creating new threads until it dies

Oh, that's very helpful to know about, ty The overwhelming majority appear to be threads in TIMED_WAITING, all waiting on the same thing: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b315cbb I've attached a screenshot which includes the stack trace. Stopping all queries

Re: Solr keeps creating new threads until it dies

Hello Can you please tell us the JVM Heap Setting for both the versions: 8.3.1, 8.9.0? I will also have to look into the following code: FileFloatSource.java:210. (will do it tonite-IST and update) Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandh

Re: Solr keeps creating new threads until it dies

CLI invocation for 8.3.1 is java -server -Xmx15826m -XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch -Xlog:gc*:file=/srv/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M -Dcom.sun.management.jmxremote -Dcom.

Re: Solr 8 autocommit vs client commits

Yes they will get committed as well. Commutwithin takes highest priority On Tue, Oct 12, 2021, 6:29 PM Saur, Alexandre (ELS-AMS) wrote: > Sorry for the noobie question guys, but I want to make sure I understand > the autocommit behaviour in Solr 8. > > Supposing I have an autocommit time of 5 mi

Re: Solr keeps creating new threads until it dies

Hello I can as of now see two changes: 1. -Xmx 2. +ExplicitGCInvokesConcurrent Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deico

Re: Solr keeps creating new threads until it dies

Yep - I edited the two instances to remove those differences and re-ran a fairly tortuous testing cycle - 100 requests/sec. Interestingly, 8.9.0 outperformed 8.3.1 significantly at this point and neither hit the threads limit or failed to create new threads. But both scaled up to well above 2k thr

UpgradeIndexMergePolicy | This policy no longer works as described | This is apparently by design

This is apparently by design. This policy no longer seems to work as described, and is actually misleading, perhaps remove it or when this policy is in effect actually stamp the indexes with the new version number? [As this is an intentional non-default configuration - my vote would be to *stam

Re: Concurrent transactions and autocommit in Solr 8

I understand your concern. I'll add more information to help clarify how the transactions are done: - The whole indexing process (client) runs in a Spark cluster. Given one document, the process of removing/inserting is done by the same Spark executor. - The pipes that perform removal/insertion

Shard leadership best practice in Solr 8

Hello, We have a Solr 8 cluster with 5 nodes and one (big) collection that is split into 5 shards. Given this scenario, what's the best way to optimize heavy indexing - splitting shard leadership amongst the nodes or have just one node being the leader of all shards? Thanks in advance!

Re: Solr keeps creating new threads until it dies

We will have to check the solr logic. Threads waiting might get reused in the future, so might be good for all you know. Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deico

Re: How to improve stats query performance?

Hi Hakan, Have you defined docValues as true for field fromid in the managed schema? If you run stats query on a field without docValues, then Solr can not make use of the OS cache and will have to load the whole index for that field into the JVM (Java virtual machine). This ca slow down the perfor

Re: Solr keeps creating new threads until it dies

Keeping threads around for re-use would certainly make sense to me, but when the code starts erroring because it's trying to create ever-more new ones rather than re-use old ones, it seems there must be a bug On Wed, 13 Oct 2021 at 15:03, Deepak Goel wrote: > We will have to check the solr logic

Re: Solr keeps creating new threads until it dies

Am sorry, did you post the thread error.I might have missed it. Can you please post it again? Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.c

NewRelic useragent and Solr memory leaks

Hi. We use newrelic to gather stats via their solr-jmx instrumentation implementation. I've noticed a memory leak with their implementation that allows metrics to be registered, but not deregistered (as MetricsManager does in Solr) I was wondering if anyone else uses newrelic and has noticed perf

Re: Solr keeps creating new threads until it dies

Sure - when solr gets pushed enough to reach the max thread count, it doesn't just re-use what it has, it keeps trying to create new ones. The logs *flood* with these errors: Oct 07 15:38:56 ip-10-40-23-148 solr[4734]: [5032072.842s][warning][os,thread] Failed to start thread - pthread_create fail

Re: Solrj for addField for indexing document

Hi I used 'type" instead of Field Type, seems to be working... Thx for the Help Closing this Thread. On 2021/10/11 17:44:38, Erik Hatcher wrote: > It appears you should use "type" instead "field type" in your Java code. > (**'type' is a required field** says the error message) > > E

Re: NewRelic useragent and Solr memory leaks

solr jmx has been a source of frustration to me for years and years. Beans come and go, names change, etc. I'm not surprised new relic leaks. The only stable monitoring is the xml stuff from the http admin api. New Relic used to have decent APM integration with solr but dropped it a few versions

Re: NewRelic useragent and Solr memory leaks

We do use NR and have also experienced performance degradation over time, but I had not gotten to the bottom of it yet. Thank you so much for this lead. Is there a public link to the ticket? On 10/13/21, 11:39 AM, "Dan Rosher" wrote: Hi. We use newrelic to gather stats via their solr

Re: NewRelic useragent and Solr memory leaks

Same here, we've been bit by NR memory leaks across several of our apps. In most cases, the solution was upgrading to the latest agent. In order to understand a bit better how this is affecting you, how did you spot the leaks caused by NR? Did you do a jmap or you have a complete snapshot of the h

Re: Solr keeps creating new threads until it dies

There could be parameters which specify: 1. How long do old threads live? And when they can be reused. 2. You might also be reaching memory restrictions for creating new threads. 3. This is a bit offbeat thinking. But is there a lot of disk activities happening as threads are nothing but files.

Re: Solr keeps creating new threads until it dies

I was gonna mention point 2 as well. Maybe you're hitting the -Xss JVM limit (256Kb) and need to manually increase it. I seem to recall that it should be at the very least 1Mb (or so it says here

Re: NewRelic useragent and Solr memory leaks

Hi All, To trigger the leak I had a script do the following: - Get old collection = collection an alias currently pointing to - Create new collection - Import to new collection - Move alias to from old collection to new collection - Delete old collection - Repeat In doing this I see, with visual

Re: Solr keeps creating new threads until it dies

Hello This might not be exactly what you are looking for, but it might give some clues: https://www.eclipse.org/lists/jetty-users/msg07971.html I think you definitely need to increase your heap size. There might be no memory-leaks, it might be just that the load which you are pushing into Solr al