Request Rate Limiters documentation unclear

Hi, When reading the Request Rate Limiters documentation, there are two things not clear to me. https://solr.apache.org/guide/solr/latest/deployment-guide/rate-limiters.html This documentation uses the term "Slot" which is a key concept here to understand how rate limits work and how to configure

Solr 9.4 - Switching _default configset to classic schema issue

Hi, I created a default-classic configset by : * copying the _default configset * rename managed-schema.xml file to schema.xml * adddeclaration in solrconfig.xml I can successfully create a collection When I test a query analysis in Admin UI with the field type "text general", the following e

Re: Slow softCommits under heavy load?

Hi, This answer comes late, but maybe it's still a topic for you. As you say you can have a lot of dynamicFields, I remember that in the past, I saw long commits due to slow segment merges related to docValues and dynamicfield. Are you intensively using DynamicFields with docValues ? Furthermore

Re: [EXTERNAL] Re: authentication for Leader/Follower replication

Hi Craig, Yes, starting with the 8.9 and 9.0 versions, Collection API allows incremental backup and much more (corruption check, backup to Amazon S3 or Google Cloud Storage) . Take a look at this umbrella JIRA https://issues.apache.org/jira/browse/SOLR-15086 Regards Dominique Le jeu. 10 août

Re: SolrJ 9.2 and Java version

Thank you. Le jeu. 13 avr. 2023 à 15:43, Shawn Heisey a écrit : > On 4/13/23 02:49, Dominique Bejean wrote: > > I don't find in documentation which minimal version is required for SolrJ > > 9.2. JAva 11 and 17 are supported for sure, but is Java 1.8 still > possible ?

SolrJ 9.2 and Java version

Hi, I don't find in documentation which minimal version is required for SolrJ 9.2. JAva 11 and 17 are supported for sure, but is Java 1.8 still possible ? Regards Dominique

Re: Solrcloud strange CPU behaviour

Le lun. 23 janv. 2023 à 19:50, Shawn Heisey a écrit : > On 1/23/23 07:38, Dominique Bejean wrote: > > On a SolrCloud 7.7 environment with 14 servers, we have one collection > with > > 1 billion documents. > > Sharding is 7 shards x 2 replicas (TLOG) > > Each

Re: Solrcloud strange CPU behaviour

On Mon, Jan 23, 2023 at 9:39 AM Dominique Bejean > wrote: > > > > Hi, > > > > On a SolrCloud 7.7 environment with 14 servers, we have one collection > with > > 1 billion documents. > > Sharding is 7 shards x 2 replicas (TLOG) > > Each solr se

Solrcloud strange CPU behaviour

Hi, On a SolrCloud 7.7 environment with 14 servers, we have one collection with 1 billion documents. Sharding is 7 shards x 2 replicas (TLOG) Each solr server hosts one replica. Indexing and searching are permanent. Suddenly one of the server has CPU usage growing during 30 minutes. Sometimes du

Re: Sharding strategy advise

Thank you Shawn. Le sam. 17 déc. 2022 à 20:39, Shawn Heisey a écrit : > On 12/17/22 04:07, Dominique Bejean wrote: > > solr.log.18:394492:2022-11-23 14:45:37.134 INFO > (zkCallback-5-thread-128) > > [ ] o.a.s.h.CdcrLeaderStateManager Received new leader state @ > :s

Re: Requests taking hours on solr cloud

Hi, May be a thread dump and a heap dump can help to find where and why this request is blocked ? May be just by finding this thread in the Solr console, you can see where the thread is blocked ? Regards Dominique Le dim. 18 déc. 2022 à 09:10, Satya Nand a écrit : > Pinging on this thread a

Re: Sharding strategy advise

Dominique Le ven. 16 déc. 2022 à 09:43, Shawn Heisey a écrit : > On 12/15/22 12:43, Dominique Bejean wrote: > > I have a sharded collection distributed over several solr nodes. Each > solr > > node hosts one shard and one replica of another shard. shards are huge > (100 &

Sharding strategy advise

Hi, I have a sharded collection distributed over several solr nodes. Each solr node hosts one shard and one replica of another shard. shards are huge (100 millions documents). Queries are using several filterQuery. filterCache for this number of documents can use high amount of heap memory. Is it

JVM threads and heap issue due to filtercache

Hi, I have a huge sharded collection. Each shard contains 100 millions docs. Queries are using at maximum 20 filter queries. 10 are very often used and the other not often. filterCache size is 10 and autoWarmCount is 10. filterCache statistics are very good except warmupTime is a little long. "C

Re: FieldCache and _version_field

's used for updates and specifically in > AtomicUpdateProcessorFactory > and UpdateLog. Presumably, if that cluster can live without atomic updates, > you can try to drop them out of update chain. > > On Tue, Dec 6, 2022 at 5:14 PM Dominique Bejean > > wrote: > > &

FieldCache and _version_field

Hi, One of my customers has a huge collection (1.5 billion docs across 14 shards). All fields are correctly configured in order to enable docValues except _version_. They are still using the old configuration with indexed=true instead of docValues and hence _version_ populate FiledCache in JVM hea

Re: Pull Interval in Pull Type replicas of Solr Cloud ?

Hi, On sematext blog, I read for TLOG replication Interval The poll time from replica to the master is set to half of the autoCommit property value or, if autoCommit is not defined, 50% of the autoSoftCommit. If both are not present it is set to 1500 milliseconds. No details for PULL replica but

Re: Advice in order to optimise resource usage of a huge server

22-10-06 2:57 AM, Dominique Bejean wrote: > > > Do not configure disks in RAID 6 but, leave 6 standard volumes (more > space > > disk, more I/O available) > > If they're running linux: throw out the raid controller, replace with > ZFS on 2 SSDs and 4 spinning rust dri

Re: Advice in order to optimise resource usage of a huge server

each > as needed > > > On Oct 6, 2022, at 6:52 PM, Dominique Bejean > wrote: > > > > Thank you Dima, > > > > Updates are highly multi-threaded batch processes at any time. > > We won't have all index in RAM cache > > Disks are SSD > > >

Re: Advice in order to optimise resource usage of a huge server

Thank you Dima, Updates are highly multi-threaded batch processes at any time. We won't have all index in RAM cache Disks are SSD Dominique Le ven. 7 oct. 2022 à 00:28, dmitri maziuk a écrit : > On 2022-10-06 4:54 PM, Dominique Bejean wrote: > > > Storage configuration is

Re: Advice in order to optimise resource usage of a huge server

g on the server) ? I note the various suggestions in your answers (ZFS, RAID 10, ...) Thank you Dima and Shawn Regards Dominique Le jeu. 6 oct. 2022 à 09:57, Dominique Bejean a écrit : > Hi, > > One of our customer have huge servers > >- Bar-metal >- 64 CPU >-

Advice in order to optimise resource usage of a huge server

Hi, One of our customer have huge servers - Bar-metal - 64 CPU - 512 Gb RAM - 6x2Tb disk in RAID 6 (so 2Tb disk space available) I think the best way to optimize resources usage of these servers is to install several Solr instances. I imagine 2 scenarios to be tested according to d

Re: Need help with DIH plugin SOLR

Hi, I suggest to take a look at Apache Nifi ETL in order to replace DIH. It can read and write into Solr, Dominique Le mer. 6 avr. 2022 à 12:44, Jan Høydahl a écrit : > Hi, > > The upcoming 9.0 release does not have DIH. And it is unclear whether the > plugin on github will be updated to work

Re: Rule-Based Authorization Plugins with JWT Authentication Plugin

security.json, by adding "principalClaim": "preferred_username" in the authentication settings. { "authentication":{ "blockUnknown": true, "class":"solr.JWTAuthPlugin", "principalClaim": "preferred_username

Re: Help with stopwords filter

Hi, Are you sure "is" is defined as a stopword at both index and query type in your analyzers ? Dominique Le lun. 4 avr. 2022 à 09:09, Arif Shaon a écrit : > Hello list, > > I am trying the following two queries, which should return the same result. > However, the first contains a stop word "i

Re: Solr as a dedicated data store?

Hi, A best practice for performances and ressources usage is to store and/or index and/or docValues only data required for your search features. However, in order to implement or modify new or existing features in an index you will need to reindex all the data in this index. I propose 2 solutions

Re: DIH and import from other core

Hi, I suggest to take a look to Apache Nifi, a great multi-threaded ETL including plugin in order to read from or write to Solr. Dominique Le jeu. 31 mars 2022 à 13:12, Tealdi Paolo a écrit : > Hi all, > > I'm searching for alternative to DIH functionality for record ingestion > from one core

Re: Search query with uppercase getting different results

Hi, I suppose, you are using the word delimiter filter with splitOnCaseChange enabled. https://solr.apache.org/guide/8_6/filter-descriptions.html#word-delimiter-graph-filter Disable this option. Dominique Le ven. 1 avr. 2022 à 05:28, ozatomic a écrit : > Thanks JAG, > > Is this something tha

Rule-Based Authorization Plugins with JWT Authentication Plugin

Hi, I am testing JWT Authentication Plugin with a keycloak IPD. Rule-Based Authorization Plugins deny access to some Solr end points. According to logs, I don't understand why. For example, in this log line we see that requested role are either "solr", "solr:admin", "consoleread" user roles are

Re: Solr 8 edismax issue with mm and q.op

a écrit : > On 3/16/22 04:52, Dominique Bejean wrote: > > I have some strange results when using both mm and q.op with edismax in > > solr 8.11. In this case it looks like q.op has priority on mm. > > > > In all other cases (solr 4 with both dismax and edismax ou

Solr 8 edismax issue with mm and q.op

Hi, I have some strange results when using both mm and q.op with edismax in solr 8.11. In this case it looks like q.op has priority on mm. In all other cases (solr 4 with both dismax and edismax ou Solr 8 with dismax) in accordance with the documentation, mm has priority on q.op. Here is my quer

Re: Issue with MMapDirectory

Hi Roman, Do you really need a 31gb heap size ? Use gceasy.io in order to analyse your heap usage. I thing server free system memory is to small for efficiente caching. I suggest to increase server memory and reduce heap size if possible. Don’t forget to disable os swap by setting swapiness to

Re: Solr heap memory

Hi, As previously said, long GC pauses should be the cause of Solr/Zookeeper communication issues. Analyse your GC logs with gceasy.io in order to confirm this. After this, you need to investigate what is causing so much heap memory consumption. Maybe you will discover misconceptions in your shema

Re: Having problem with implicit field "_root_"

Hi, I suggest you define both id and _root_ fields as string, and you populate id with uuid generated by UUIDUpdateProcessor in an updateRequestProcessorChain. See Solr Wiki - https://cwiki.apache.org/confluence/display/solr/uniquekey Dominique Le mar. 31 août 2021 à 22:30, Ed Yu a écrit :

Re: OutofMemory Error in solr 6.5

t > any issue, So I believe cache size should not be an issue. If the cache > size had been the culprit, the issue would have been frequent. isn't it? > > > > On Mon, Aug 9, 2021 at 6:44 PM Dominique Bejean > wrote: > >> Hi, >> >> You don't provide

Re: Cpu 100%

Ok, I got it. You mean job that send data to Solr for indexing is single thread, not Solr itself :) Dominique Le lun. 9 août 2021 à 17:19, Deepak Goel a écrit : > > https://hybrisarchitect.com/solr-configuration-increase-number-of-indexer-threads/ > > On Mon, 9 Aug 2021, 18:04 Domi

Re: OutofMemory Error in solr 6.5

Hi, You don't provide information about the number of documents. Anyway, all your cache size and mostly initial size are big. Cache are stored in JVM heap. About cache size, most is not always better. Did you make some performance benchmarks in order to set these values ? Try with the default va

Re: Cpu 100%

Hi Jeff, How many CPU ? What is the CPU load average (information provided by Linux top command in the first line) ? I don't think it is the root cause of your issue, but 1 NRT + 1 TLOG replica per shard is not a usual practice. You have to create 2 TLOG replicas per shard and let Solr elect the

Re: Cpu 100%

Hi Deepak, Can you explain Solr single-threaded mode ? I am not aware of any settings about a single-threaded mode ! Regards Dominique Le lun. 9 août 2021 à 07:16, Deepak Goel a écrit : > Looks like your SolrCloud is running in a single-threaded mode. So adding > CPU's might not help! > > >

Re: Rule-Based Authorization Plugin not working as described in the documentation

Hi, I think this is related to this issue https://issues.apache.org/jira/browse/SOLR-13097 Regards Dominique Le jeu. 15 juil. 2021 à 10:37, Thomas Corthals a écrit : > The docs are technically correct in consistently talking about "collection" > and not "core" in this regard. They could make

Re: Request help / steps to upgrade from apache solr 7.7.3 to 8.5.2

Hi Prashanth, Please consider upgrading JDK to version 11. If all your fields are stored, maybe you can write a script in order to dump your data to json files and so reindex them in Solr 8. If your indexes were created with Solr 7, you can try the IndexUpgraderTool https://solr.apache.org/guide/

Re: SolrException : java.lang.OutOfMemoryError: Java heap space

or when this occurs, and then never after and no oom error ever > again > > > On Jun 25, 2021, at 12:47 PM, Dominique Bejean < > dominique.bej...@eolya.fr> wrote: > > > > Hi Dave, > > > > I agree with not allocating more than 31 GB for Xms/Xmx as it

Re: SolrException : java.lang.OutOfMemoryError: Java heap space

Hi Dave, I agree with not allocating more than 31 GB for Xms/Xmx as it is the upper limit in order for the JVM to use compressed oops ( https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/), but why directly 31 Gb without more Solr usage analysis ? Dominique Le

Re: tlog size issue- solr cloud 6.6

:solrcollection s:shard2 r:core_node1 >> x:solrcollection_shard2_replica1] >> org.apache.solr.update.DirectUpdateHandler2; start >> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} >> >> >> On S

Re: Replication and Score Issue

Hi, If your replicas are all NRT, they both index documents. Their commit and segment merge cycles are independant and so yes, see different MaxDoc and DeletedDoc for each replicas is normal. We can expect BM25 doesn't care about deleted docs, but I can't answer with certainty. Regards. Dominiq

Re: tlog size issue- solr cloud 6.6

may increase softcommit timeout but as per my understanding it only > helps in searching availability prior to commit to indexes. > > On Sun, 21 Mar 2021 at 4:29 AM, Dominique Bejean < > dominique.bej...@eolya.fr> wrote: > >> Hi, >> >> By heavy query, do you

Re: tlog size issue- solr cloud 6.6

Hi, By heavy query, do you mean you have a high query rate and/or you need index update be available fast after few seconds (NRT search) ? Do you see the hard commits in logs ? Can you try to increase autoSoftCommit to 30 seconds or more ? Regards Dominique Le sam. 20 mars 2021 à 18:53, Ritvik