Limits + locks when multiple clients /update at once

* Are there any constraints as to how to safely call the /update handler for a core from multiple clients? * Are there locking issues we should be aware of? * Is there any way multiple /update calls can corrupt a core? Backstory: Back in 2013, when we first started using Solr 4.2.0, we had pro

How can I performance-tune my warming queries?

I’m trying to figure out why my warming is taking so long. It’s taking about 20-40 seconds on average. Can I measure where it’s spending its time? I’ve got my firstSearcher and newSearcher set up like this: ... world popular_score desc, grou

Re: How can I performance-tune my warming queries?

Thanks very much for this. This is a huge help. > What I would recommend is that (at a time when query traffic is lowest) you > turn off all warming, restart, and then do some manual queries where you > check each fq and facet individually. Rebooting or clearing the OS disk > cache before ea

Re: How can I performance-tune my warming queries?

> On Oct 18, 2021, at 2:38 PM, Shawn Heisey wrote: > >> What should we have in the newSearcher startup query, if the new searcher is >> going to bring over the cached FQs from an existing searcher? > > I know that filterCache handles autowarming for fq parameters. I do not know > whether qu

First cross-join faceting after a commit is slow

I’m on Solr 8.10.1 and having a performance problem with my cross-core join facets. Here’s my basic query, with the interesting parts bolded (the joins in two facet.query fields) curl "$URL" --silent --show-error \ -X POST \ -F "q=($word AND -parent_tracings:($word))" \ -F 'df=titl

Re: NRT Searching (getting it working)

> > I'm not sure how to understand the solr_gc.log file (but I'd like to) There’s a product called gceasy at gceasy.io . You can get a basic report on your GC log from uploading your log to them for analysis. Andy

Re: NRT Searching (getting it working)

> On Nov 17, 2021, at 12:41 PM, Derek C wrote: > > That's an amazing online tool - thanks Andy It was Shawn Heisey who pointed me to it. There many other JVM GC tools out there if you search a bit. https://sematext.com/blog/java-gc-log-analysis-tools/

Re: Searcher and autoSoftCommits + softCommit

> > You were spot on, commitWithin was being set on each commit. I was able to > verify by temporarily turning on debug logging for DirectUpdateHandler2 What did you do to enable that? I didn’t know you could do such a thing. Is it specific to that handler? Thanks Andy

Is the showItems argument for fieldValueCache used for anything?

It looks to me like the showItems argument for the fieldValueCache is not used. I can’t find any documentation of it, although it was mentioned in the changelog for v8.1.0. I looked through the source and I can’t see where the value is used. It gets a default value, but never seems to be used,

Re: Is the showItems argument for fieldValueCache used for anything?

> On Dec 3, 2021, at 9:39 AM, Mikhail Khludnev wrote: > > It seems it's gone https://issues.apache.org/jira/browse/SOLR-15762 > > I will miss showItems, it was really useful a long ago. I don’t understand how that ticket relates. I don’t see

Re: Not able to write solr logs in json format

> On Dec 3, 2021, at 9:55 AM, Kakolu, Karthik > wrote: > > Trying to write solr.log in json format but unsuccessful. How exactly are you unsuccessful? Do you get logs but they aren’t in JSON as expected? If so, what format are they in? Are you able to write logs in other formats than JSON

Re: multiple values encountered for non multiValued field Solr 8.10.1

> We are trying to index documents for a collection. This worked in Solr 3.6.1, > but running this search under Solr 8.10.1 generates the below error. > > multiple values encountered for non multiValued field > > Any help would be appreciated, and feel free to ask if you need further info. We

Re: 0-day Apache log4j RCE vulnerability

I trust that by now you’ve seen the discussion earlier today on this mailing list about it.

Re: Zookeeper and Solr and CVE-2021-44228

> On Dec 13, 2021, at 8:20 AM, Michael Conrad wrote: > > I presume this also needs fixing for zookeeper nodes? Anything that logs with log4j.

Re: [ANNOUNCEMENT] Solr's Docker images were updated to remediate a CVE

For those of you like me who want to explicitly set the variable without relying on which of the two Docker images with the same tag you’re pulling down, and you’re using a Dockerfile to add on to make your own Solr Docker image, add these lines: # Add option to mitigate log4j security vulnerab

Re: When will solr 8.11.1 become available?

> It is impossible to give you an accurate prediction for the release date of > Solr 8.11.1. It sounds like it’s safe to say that the release will be “on the order of at least a week from now” would be safe, right? That might be all the accuracy that someone needs.

Re: Question Apache Solr 7.7.0, 8.7 and 8.9 - log4j vulnerability

> On Dec 14, 2021, at 9:00 AM, Manisha Rahatadkar > wrote: > > We are using Apache Solr 7.7.0, 8.7 and 8.9 on Windows and Linux > environment. What mitigation option do we need to take for this vulnerability? https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-20

Re: Log4J saga (CVE-2021-45046)

> > Is there already an Idea when 8.11.1 is supposed to be released ? This was discussed yesterday. Check the archives for the full explanation. Short version: can’t give a definite date but it will be no sooner than a week from now.

Log4j remediation in the Docker image

> On Dec 16, 2021, at 8:26 AM, Carlos Cueto wrote: > > Any idea when it will be available on Docker Hub? 8.11.1 tag is still not > added. I don’t know, but yesterday I went and changed my build process for our Docker image of Solr to delete the JNDI classes from the jar files as a stopgap un

Re: Question about unintended deletions of Solr documents

Can you please post specific queries that you are trying? Cut and pasted, not paraphrases. > On Dec 18, 2021, at 3:13 PM, Claire Burke wrote: > > If I enter a query in the q field (which is associated with the > Request-Handler (qt) type /select), then I enter a delete query (which is > ass

Re: reloading all the cores

> Is there an http request that can make all the cores on a solr server > reload? I don’t think there is, but you can use the STATUS API call (https://solr.apache.org/guide/8_11/coreadmin-api.html ) to get a list of all the cores so that yo

Re: Problem with Join query and FilterCache

> On Jan 20, 2022, at 8:02 AM, Mike Drob wrote: > > Yep, you should change from LRUCache to CaffeineCache in your solrconfig.xml And, the CaffeineCache has to be set as async. It defaults to async, but I added the async=“true” to be explicit in my solrconfig.xml

Re: SOLR 8.11.1 :: VELOCITY :: Can't access JAVA-object's static methods

> On Feb 3, 2022, at 3:03 AM, Jan Høydahl wrote: > > This is/was a security hole and a big anti-pattern. Is this still possible in 8.x? If so, I think it would be worth putting in the docs that it can be a security problem. I can probably do that. Andy

Re: Vulnerability on solr port

> On Feb 14, 2022, at 3:35 AM, Anchal Sharma2 > wrote: > > We have got following vulnerability on port where apache solr is running on > few of our servers .Does anyone have any ideas/suggestions on how to mitigate > this ? > Vulnerability -> Web Server HTTP Heade

Re: Vulnerability on solr port

> On Feb 14, 2022, at 3:35 AM, Anchal Sharma2 > wrote: > > We have got following vulnerability on port where apache solr is running on > few of our servers .Does anyone have any ideas/suggestions on how to mitigate > this ? > Vulnerability -> Web Server HTTP Heade

Re: Solr 8.6.2 - download full data

> On Feb 25, 2022, at 8:19 AM, Anuj Bhargava wrote: > > I tried with 1 records. Need to download all > http://xx.xxx.xxx.xxx:8983/solr/data_2019/select?q=*%3A*&rows=1&wt=csv > Use a command-line utility like c

Re: Solr 8.11.1 upgrading LOG4J from 2.16 to 2.17

> On Mar 23, 2022, at 1:36 PM, Heller, George A III CTR (USA) > wrote: > > Can someone tell me where I can download an upgrade or patch for LOG4J and > instructions on how to implement it? > See https://solr.apache.org/security.html

Re: Solr 8.11.1 upgrading LOG4J from 2.16 to 2.17

Go to the https://solr.apache.org/security.html URL and you will find instructions there on what to do. Andy

Re: Problem with facet in SOLR

> I have indexed 4 fields and want to use facet on "taxo_domain_mother" but i > am not getting any result It looks like you don’t have facet.field=taxo_domain_mother specified in your query. It’s hard to tell exactly because screenshots make it difficult to figure things out. > Do i need t

Re: Problem with facet in SOLR

> On Apr 1, 2022, at 3:59 AM, Neha Gupta wrote: > > Now i had set docValues=true for this attribute and not facet is working. > > Just want to know is it necessary to set the docValue to true to make facet > working for the attribute? I don’t think docValues is necessary, but it will make th

Re: Regarding maximum number of documents that can be returned safely from SOLR to Java Application.

> On Apr 27, 2022, at 3:23 PM, Neha Gupta wrote: > > Just for information I will be firing queries from Java application to SOLR > using SOLRJ and would like to know how much maximum documents (i.e maximum > number of rows that i can request in the query) can be returned safely from > SOLR.

Re: Regarding maximum number of documents that can be returned safely from SOLR to Java Application.

> > So my question is if i request in one request lets say approximate 10K > documents using SOLRJ will that be OK. By safe here i mean approx. maximum > number of documents that i can request without causing any problem in > receiving a response from SOLR. I’m still not clear what you’re ask

Re: Regarding maximum number of documents that can be returned safely from SOLR to Java Application.

> 1. Why do you need to return so many search results at the same time? If > it's a typical search usecase, could you not work with some manageable list > of documents, say 50/100? But I'm guessing this is not a typical search > that you're planning to support. I’d just like to point out that Neh

Collapsing on a field works, but expand=true does nothing

I’m working on the collapse/expand functionality. I query for books, and I want to collapse my search results on their tf1flrid, basically a family ID. I also want to do an expand so I can see what titles were collapsed out. I’m looking at the docs here: https://solr.apache.org/guide/8_11/col

Re: Collapsing on a field works, but expand=true does nothing

> Is it possible that the expand component isn't registered in your > deployment? The expand component is a default component but have you > overridden the defaults? Yes, that’s exactly what happened. Turns out that I had pulled out the unused handlers. Thanks, Andy

Re: Solr compatibility with Oracle Database 19c Database

> On Jun 8, 2022, at 2:35 PM, Yennam, M wrote: > > We are currently using Solr 4.9.0 which is connecting Oracle 12cR1 and we are > planning to upgrade our Database to Oracle 19c. So, the question that I have > is – Is SOLR 4.9.0 compatible with Oracle 19c, if not what is the minimum > versio

Re: Solr compatibility with Oracle Database 19c Database

> some folks who care enough to contribute fixes to it. Using another tool or > custom code to query the database and submit updates via the solr JSON api > or SolrJ client is currently recommended over DIH. That’s why I had to write a tool to do the exporting from Oracle, massaging into JSON, an

Re: Solr faceting

> I would like to know if solr adds faceting fields by default when we do any > search. We have an id field which is still coming up in field cache even > after it is removed from sorting and faceting queries in the code. > Therefore, I would like to know if solr is adding this by any chance. N

Re: Solr faceting

> On Jul 18, 2022, at 3:11 AM, Poorna Murali wrote: > > There is a solr search api which does not have either sorting or faceting > done with id field. But, after we execute the API, we do see id field entry > in field cache. I checked the solrconfig file too, we have not added any > id confi

Re: Retain Data Import Handler In Solr9.0

> On Jul 22, 2022, at 1:19 PM, dmitri maziuk wrote: > >> The DIH does not yet support Solr 9 but I don't think it'll be long before >> it does. > > FWIW I've been gradually migrating our DIH imports to little python scripts; > with all the extra things you can do in those, and less bloat in t

Re: Retain Data Import Handler In Solr9.0

> On Jul 22, 2022, at 1:39 PM, Dave wrote: > > Oh look into perls fork manager module, > > https://metacpan.org/pod/Parallel::ForkManager > I’m aware of the numerous tools like that (I’ve been doing Perl since the 90s https://metacpan.or

Re: Fastest way to index data to solr

> On Sep 29, 2022, at 4:17 AM, Jan Høydahl wrote: > > * Index with multiple threads on the client, experiment to find a good number > based on the number of CPUs on receiving side That may also mean having multiple clients. We went from taking about 8 hours to index our entire 42M rows to a

Re: Fastest way to index data to solr

I can’t imagine a case where the speed in parsing the input data won’t be dwarfed by the time spent on everything else. You’re talking about an in-memory operation that does a ton of I/O. It’s not going to make a noticeable difference once way or the other. > I have a followup question. Is JS

Re: HTTP errors POSTing to 8.11.2

> On Oct 27, 2022, at 5:44 PM, dmitri maziuk wrote: > > has anyone gone through the exercise of replacing Data Import Handler with > scripts that POST JSON and if so, are your scripts still working OK with > 8.11.2? That's exactly what I've done a couple of years ago and they work just fine

Re: Using the fq parameter to filter for a value that is multivalued field.

> On Dec 9, 2022, at 11:22 AM, Matthew Castrigno wrote: > > "myField":["apple, pear"] That's not multivalued. That's a single value and the value is "apple, pear". You need to pass multiple values to Solr for the field when you do your indexing. Basically, you need to pass one myField:ap

Re: Solr Query time performance

> On Jan 29, 2023, at 4:45 AM, marc nicole wrote: > > Let's say you're right about the 200 rows being too few. From which row > count I can see the difference reflected in the results as it should (Solr > faster)? It depends on how much data is in each record, but I'd think 10,000 - 100,000

Re: Query time

Please include your schema and some sample queries so we have specifics to go on. > On Feb 8, 2023, at 9:00 AM, Mike wrote: > > I have a standalone Solr server and an index of millions of documents. > Some queries that e.g. more than 1 million times exist takes a long time. > I only need the

Re: SOLR security scan question

> Any news on this? > > We know some of them are covered in > https://solr.apache.org/security.html#cve-reports-for-apache-solr-dependencies > but not all. > We have also seen the > https://lists.apache.org/thread/539bkq8r11msjpl3yo1ssvy77kmdrps7 > Can we have a resolution for the above? What

Re: join query parser performance

> On May 25, 2023, at 7:51 AM, Ron Haines wrote: > > So, when this feature is enabled, this negative &fq gets added: > -{!join fromIndex=primary_rollup from=group_id_mv to=group_member_id > score=none}${q} Can we see collection definitions of both the source collection and the join? Also, a

Re: End of Life of Solr 8.11?

Do we have a rough idea of when we think Solr 10 will be? A few months? A year? I just have an upgrade project to Solr 9 on the horizon and might hold off on it a bit if Solr 10 is imminent. > Yes, it is EOL the same date that 10.0 is shippped. The release date of 10.0 > is yet to be determine

Re: Solr Crawl Error

I don't know what tool you're using to do the crawling, because Solr itself does not crawl. It just indexes text. So you must have some other tool that is fetching data and feeding it to Solr. > WARNING: Solr returned an error #404 Not Found > WARNING: IOException while reading response: java.

Re: Performance of solr 9.3 vs 8.11

> On Oct 12, 2023, at 12:54 PM, Natarajan, Rajeswari > wrote: > > Does anyone see any query performance degradation from 8.11 to 9.3. Please > let me know What is it that you are really asking? Are you wondering if there will be a slowdown if you upgrade from 8.11 to 9.3? If so, we need

Re: How to do fastest loading and indexing

> On Nov 12, 2023, at 9:16 AM, Vince McMahon > wrote: > > So, if I split the single cvs into two and using two programs sending each > of the splits, Solr will handle the parallel loading with multiple > threads. I don't have to make changes to Solr, right? Yes, that's correct. We were lo

Re: facet query question

> Does Solr have something caching results for facet queries over large > dataset? Is there example how to make facet query faster? Yes. There are many articles about query caching in Solr, plus the docs. https://solr.apache.org/guide/8_8/query-settings-in-solrconfig.html for one version

Re: Will solr support in AWS/Azure Cloud platform

> Our application moving to cloud... So all the components including solr we > planning to move to cloud. > > Can we able to install Solr in AWS/Azure Cloud platform, anybody did that > before. Yes, you can run Solr on cloud platforms. Yes, people have done that. "Cloud" just means that it's a m