Re: timeAllowed in Solr 9

2024-11-11 Thread Dominic Humphries
time-limit properly. > > I'm going to presume that the details of your analysis are proprietary, and > that you possibly want to sort that out internally or with a consultant, > but I am interested in whether or not this exact query behaved similarly on > 8.11 or not, and if the

Re: timeAllowed in Solr 9

2024-11-11 Thread Dominic Humphries
) 24.465s AND+NOT+(internship+OR+intern+OR+graduate+OR+welder) 2.062s AND+NOT+(internship+OR+intern+OR+welder+OR+graduate)' 1.722s AND+NOT+(internship+OR+graduate+OR+intern) 25.353s AND+NOT+(internship+OR+graduate+OR+welder) 24.473s On Mon, 11 Nov 2024 at 14:03, Dominic Humphries wrote: > Hm

Re: timeAllowed in Solr 9

2024-11-11 Thread Dominic Humphries
ym(description:apprentic description:apprenticeshi p description:intern description:internship))^2.0 | (Synonym(title:apprentic title:apprenticeship title:intern title:internship))^5.0)~0.01 ((Synonym(description:grad description:graduat))^2.0 | Synonym(company:grad c

Re: timeAllowed in Solr 9

2024-11-08 Thread Dominic Humphries
out some other option.. On Fri, 8 Nov 2024 at 14:01, Gus Heck wrote: > The mailing list usually strips out attachments. You'll need to paste it > into the body of the email. > > On Fri, Nov 8, 2024 at 7:16 AM Dominic Humphries > > wrote: > > > Fair enough! See att

Re: timeAllowed in Solr 9

2024-11-08 Thread Dominic Humphries
names or sensitive information with "" or similar. > > On Thu, Nov 7, 2024 at 12:21 PM Dominic Humphries > wrote: > > > Yes, sorry, not cloud, afaik it's single-sharded. > > > > Same query with facet fields removed takes just as long to run. Adding > the &g

Re: timeAllowed in Solr 9

2024-11-07 Thread Dominic Humphries
;size":"20.55 GB" On Thu, 7 Nov 2024 at 13:27, Gus Heck wrote: > This is interesting, can you give us a feel for the size/structure of the > index (# of documents, size of index, # of shards)? > > On Thu, Nov 7, 2024 at 7:52 AM Dominic Humphries > > wrote: > > &

Re: timeAllowed in Solr 9

2024-11-07 Thread Dominic Humphries
facet.field=contract_time&facet.field=%7B!key%3Dcompany%3Aid%7Dcompany_id&f.description.hl.fragsize=180&hl=false&rows=20&start=0&q=(carroll_county+OR+Aldi+OR+Cashier+OR+Kohls)+AND+NOT+(internship+OR+intern+OR+graduate)&fq=location_id%3A151946&fq=boosted%3A1&fq=%7B!

Re: timeAllowed in Solr 9

2024-11-07 Thread Dominic Humphries
g cloud/zookeeper? > > The request you showed has a lot of facets on it. How much difference does > it make to the situation if you just send the query without the facets? > > Also add &debug=query and send us the debug output from the header when you > do that... > > &g

Re: timeAllowed in Solr 9

2024-11-06 Thread Dominic Humphries
I spoke too soon, I figured out how to get VisualVM talking to solr. Now I'm just not sure what to do with it - what sorts of things am I looking for? On Wed, 6 Nov 2024 at 16:40, Dominic Humphries wrote: > Unfortunately I don't know Java anywhere near well enough to know my w

Re: timeAllowed in Solr 9

2024-11-06 Thread Dominic Humphries
ng would be to provide a minimum working example of the > problem you are experiencing. > > On Wed, Nov 6, 2024 at 9:55 AM Dominic Humphries > > wrote: > > > I've tried both timeAllowed and cpuAllowed and neither are restricting > the > > amount of time the que

timeAllowed in Solr 9

2024-11-06 Thread Dominic Humphries
Hi folks, we're testing Solr 9.7 to upgrade our existing 8.11 stack. We're seeing a problem with long requests: we send `timeAllowed=4900` which works fine on the existing 8.11 and keeps requests to just a few seconds. With 9.7, however, the flag is basically ignored - requests can take over 30 s

Re: timeAllowed in Solr 9

2024-11-06 Thread Dominic Humphries
Particularly > important is any information about where (what code) your queries are > spending a lot of time in if you have it. > > On Wed, Nov 6, 2024 at 6:18 AM Dominic Humphries > > wrote: > > > Hi folks, > > > > we're testing Solr 9.7 to upgrade our

Testing with vectors

2024-08-28 Thread Dominic Humphries
We use Vegeta for load-testing new versions before we upgrade to them. It's worked well for us, as it allows us to save thousands of queries from live and re-run them on our test instances for realistic tests. We're now trying to test vector searches, tho, and it's a bit of a problem: If we put th

Vector search core size/disk usage

2024-08-21 Thread Dominic Humphries
We've been testing using vector search in 9.4 recently, and whilst the quality of searches is looking good, we're concerned by what we're seeing in core size/disk use. The core we're testing is typically around 2.5GB in size. In the test instance with vectors added, it's typically hovering around

Solr upgrades perform badly: The final resolution!

2022-03-08 Thread Dominic Humphries
Hey folks, you might remember me posting a while back about a problem we were having with upgrading: We were using 8.3 with no major problems, but when we tried upgrading to the latest Solr we saw slower response times and increased failures, to the point where we couldn't even consider upgrading.

Re: Return field with astrick(*) is going to slow in Solr 8.9.0

2021-11-12 Thread Dominic Humphries
It's not exactly an answer, but we've also experienced slowness when trying to upgrade. Our testing showed we started seeing degraded performance from v8.6 - if 8.5.2 would be "new enough" for you, it may be worth testing performance with that version to see if it's any better. On Fri, 12 Nov 2021

Re: Significant performance hit on replication compared to older version

2021-11-05 Thread Dominic Humphries
ularly performant anyway. There doesn't > seem to have been any activity around EFF between those versions apart > from some doc fixes > > https://issues.apache.org/jira/browse/SOLR-14968?jql=text%20~%20%22externalfilefield%22 > > > Hope some of these links help you track d

Re: Significant performance hit on replication compared to older version

2021-10-27 Thread Dominic Humphries
Very good question, for which I currently have no answer! On Wed, 27 Oct 2021 at 17:15, Deepak Goel wrote: > Why wouldnt the performance hit not happen for 8.3.1? > > On Wed, 27 Oct 2021, 20:07 Dominic Humphries, > wrote: > > > At last, I think we've got it! > &

Re: Significant performance hit on replication compared to older version

2021-10-27 Thread Dominic Humphries
Thanks all for your help! Dominic On Tue, 26 Oct 2021 at 15:31, Dominic Humphries wrote: > No problem, I've been trying to get my head around how it all works myself! > > As per > https://solr.apache.org/guide/8_9/working-with-external-files-and-processes.html > our sch

Re: Significant performance hit on replication compared to older version

2021-10-26 Thread Dominic Humphries
; Charlie > > On 25/10/2021 17:11, Dominic Humphries wrote: > > Performance with the replica pulling from 8.3.1 was actually worse. And > > looking at the data in the databases and the boost file contents, I'm > > dubious it's a problem of incompatible boost files. I thi

Re: Significant performance hit on replication compared to older version

2021-10-25 Thread Dominic Humphries
or the issue we see. Not sure what else to test to verify or disprove this.. On Mon, 25 Oct 2021 at 14:56, Dominic Humphries wrote: > I think I found it! > > I didn't realise, but we have boost files for the core I'm testing and the > boost is applied after replication!

Re: Significant performance hit on replication compared to older version

2021-10-25 Thread Dominic Humphries
> > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.makeinindia.com/home > > > On Thu, Oct 21, 2021 at 9:57 PM Dominic Humphries > wrote: > > > One mo

Re: Significant performance hit on replication compared to older version

2021-10-21 Thread Dominic Humphries
ol > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.makeinindia.com/home > > > On Thu, Oct 21, 2021 at 9:57 PM Dominic Humphries > wrote: > > > One more tidbit: I just tried leaving replication off

Re: Significant performance hit on replication compared to older version

2021-10-21 Thread Dominic Humphries
One more tidbit: I just tried leaving replication off for a few hours and then triggering a "big" replication run so I could see the distinct stages. - Beginning replication didn't cause any performance degradation. - Several minutes of downloading the replication files saw no degradation

Significant performance hit on replication compared to older version

2021-10-21 Thread Dominic Humphries
My adventure to figure out why we have significantly worse performance on 8.9.0 compared to 8.3.1 continues. As mentioned in a previous thread, when running tests we're seeing around 90% success rates with 8.9 compared to around 98% with 8.3.1 Running tests at 1min for better granularity, we saw

Re: Solr keeps creating new threads until it dies

2021-10-19 Thread Dominic Humphries
eed to find out what. > > > Deepak > "The greatness of a nation can be judged by the way its animals are treated > - Mahatma Gandhi" > > +91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/de

Re: Solr keeps creating new threads until it dies

2021-10-18 Thread Dominic Humphries
91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.makeinindia.com/home > > > On Mon, Oct 18, 2021 at 9:04 PM Dominic Humphries

Re: Solr keeps creating new threads until it dies

2021-10-18 Thread Dominic Humphries
w long do old threads live? And when they can be reused. > > > > > > 2. You might also be reaching memory restrictions for creating new > > threads. > > > > > > 3. This is a bit offbeat thinking. But is there a lot of disk > activities > > > happ

Re: Solr keeps creating new threads until it dies

2021-10-13 Thread Dominic Humphries
d by the way its animals are treated > - Mahatma Gandhi" > > +91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.

Re: Solr keeps creating new threads until it dies

2021-10-13 Thread Dominic Humphries
ook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.makeinindia.com/home > > > On Wed, Oct 13, 2021 at 6:05 PM Dominic Humphries > wrote: > > > Yep - I edited the two insta

Re: Solr keeps creating new threads until it dies

2021-10-13 Thread Dominic Humphries
o Green" > > Make In India : http://www.makeinindia.com/home > > > On Wed, Oct 13, 2021 at 5:09 PM Dominic Humphries > wrote: > > > CLI invocation for 8.3.1 is > > java -server -Xmx15826m -XX:+UseG1GC -XX:+PerfDisableSharedMem > > -XX:+ParallelRefProcE

Re: Solr keeps creating new threads until it dies

2021-10-13 Thread Dominic Humphries
be judged by the way its animals are treated > - Mahatma Gandhi" > > +91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.ma

Re: Solr keeps creating new threads until it dies

2021-10-13 Thread Dominic Humphries
that to determine > what all those threads are doing and where they are getting stuck. You can > post parts of the thread dump back to this email thread as well. > > > > Joel Bernstein > http://joelsolr.blogspot.com/ > > > On Tue, Oct 12, 2021 at 11:15 AM Dominic

Solr keeps creating new threads until it dies

2021-10-12 Thread Dominic Humphries
We run 8.3.1 in prod without any problems, but we're having issues with trying to upgrade. I've created an 8.9.0 leader & follower, imported our live data into it, and am testing it via replaying requests made to prod. We're seeing a big problem where fairly moderate request rates are causing the

Re: Slow GC causing high heap usage

2021-09-03 Thread Dominic Humphries
settings (heap and GC)? > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > On Sep 2, 2021, at 8:28 AM, Dominic Humphries > wrote: > > > > We're trying to upgrade from 8.3.1 to 8.8.1 but my pre-releas

Re: Slow GC causing high heap usage

2021-09-03 Thread Dominic Humphries
, 2 Sept 2021 at 16:52, Shawn Heisey wrote: > On 9/2/2021 9:28 AM, Dominic Humphries wrote: > > We're trying to upgrade from 8.3.1 to 8.8.1 but my pre-release testing > has > > shown us some performance issues. Examination of the GC log shows that > the > > pos

Slow GC causing high heap usage

2021-09-02 Thread Dominic Humphries
We're trying to upgrade from 8.3.1 to 8.8.1 but my pre-release testing has shown us some performance issues. Examination of the GC log shows that the possible cause may be here: 8.3.1 graphs: https://imgur.com/a/ZM9wdob 8.8.1 graphs: https://imgur.com/a/UzMinwJ The test cycle here is 2 mins with

"The request took too long" cause exceptions in upgrade testing

2021-08-31 Thread Dominic Humphries
I'm trying to upgrade from 8.3.1 to 8.8.1, and we're seeing slower performance and higher rate of failed requests when testing the upgrade. The main culprit seems to be when we're burdening the service enough to start causing "The request took too long to iterate over doc/point values" warnings. O