Did you also run this against 8.11 and get shorter times? It has always been possible for timeAllowed to overrun somewhat (the code has to periodically check if limits have been exceeded, and the work between checks isn't going to stop), but if how much it overruns has changed drastically we do want to look into that.
On Thu, Nov 7, 2024 at 11:46 AM Darren Foreman <dar...@adzuna.com.invalid> wrote: > Hi all, > > I also have little java expertise, but I seem able to replicate the problem > locally like this: > > 1. Downloaded a large CSV: the UK government's "price paid data" CSV for > an entire year: > > http://prod.publicdata.landregistry.gov.uk.s3-website-eu-west-1.amazonaws.com/pp-2019.csv > 2. Save the above in a directory called solrdata, and edit it > 1. swap the first two lines around (so that type inference works when > importing) > 2. add a header to it > > > > uuid,price,transfer_date,postcode,property_type,old_new,duration,paon,saon,street,locality,town_city,district,county,ppd_category,record_status > > 3. Run solr in a container with constrained resources - 1 CPU, 512mb > memory > > docker run --rm --cpus 1 --memory 536870912 -v "$PWD/solrdata:/var/solr" > -p 8983:8983 --name solr solr:9.7.0 > > 4. Create a core that will infer the schema from data > > docker exec -it solr ./bin/solr create -c landreg -n > data_driven_schema_configs > > 5. Import the CSV. I called mine landreg-data.csv: > > docker exec -it solr ./bin/post -c landreg /var/solr/landreg-data.csv > > 6. Query for a deep-paged set of results after applying a sort, > supplying timeAllowed: > > curl > > > 'localhost:8983/solr/landreg/select?fl=uuid&q=*:*&sort=transfer_date+desc&start=940000&debug=timing&timeAllowed=200' > > > For me, the timeAllowed parameter is almost always seemingly ignored i.e. > exceeded (especially if low) - here I ask for 200ms, or even 2000ms: > > $ curl -s > > 'localhost:8983/solr/landreg/select?fl=uuid&q=*:*&sort=transfer_date+desc&start=940000&debug=timing&timeAllowed=200' > \ > > | jq .debug.timing.process.query > { > "time": 3954.0 > } > $ curl -s > > 'localhost:8983/solr/landreg/select?fl=uuid&q=*:*&sort=transfer_date+desc&start=940000&debug=timing&timeAllowed=2000' > | jq .debug.timing.process.query > { > "time": 3645.0 > } > > > On 2024/11/06 15:34:58 Gus Heck wrote: > > If you have access to a test instance where the problem can be > reproduced, > > attaching a profiler would be one way. Another cruder method is to use > > jstack to dump all the threads. > > > > Another way to tackle this is to help us reproduce your problem. Can you > > share details about your query? Obviously, please don't post anything > your > > company wouldn't want public, but if you can share some details that > would > > be a start. > > > > The ideal thing would be to provide a minimum working example of the > > problem you are experiencing. > > > > On Wed, Nov 6, 2024 at 9:55 AM Dominic Humphries > <do...@adzuna.com.invalid > > > > wrote: > > > > > I've tried both timeAllowed and cpuAllowed and neither are restricting > the > > > amount of time the queries take to run. I have a test query that's > reliably > > > taking 20-30 seconds, if there's any useful debug params or such I can > run > > > to provide the information you want I'm happy to run them - I'm not > sure > > > how to usefully interrogate solr for where its time is being spent, > sorry > > > > > > Thanks > > > > > > On Wed, 6 Nov 2024 at 14:25, Gus Heck <gu...@gmail.com> wrote: > > > > > > > There are unit tests that seem to suggest that timeAllowed still > works, > > > can > > > > you provide some more information about your use case? Particularly > > > > important is any information about where (what code) your queries are > > > > spending a lot of time in if you have it. > > > > > > > > On Wed, Nov 6, 2024 at 6:18 AM Dominic Humphries > > > > <do...@adzuna.com.invalid> > > > > wrote: > > > > > > > > > Hi folks, > > > > > > > > > > we're testing Solr 9.7 to upgrade our existing 8.11 stack. We're > > > seeing a > > > > > problem with long requests: we send `timeAllowed=4900` which works > fine > > > > on > > > > > the existing 8.11 and keeps requests to just a few seconds. > > > > > > > > > > With 9.7, however, the flag is basically ignored - requests can > take > > > over > > > > > 30 seconds whether the flag is present or not, which is causing > higher > > > > CPU > > > > > load and slowing response times. > > > > > > > > > > I've tried setting the flag suggested in > > > > > > > > > > > > > > > > > > > https://solr.apache.org/guide/solr/latest/upgrade-notes/major-changes-in-solr-9.html#use-of-timeallowed > > > > > - but even with solr.useExitableDirectoryReader set we still don't > get > > > > the > > > > > desired behaviour. > > > > > > > > > > Is there anything else I can try to get the old behaviour back? > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > -- > > > > http://www.needhamsoftware.com (work) > > > > https://a.co/d/b2sZLD9 (my fantasy fiction book) > > > > > > > > > > > > > -- > > http://www.needhamsoftware.com (work) > > https://a.co/d/b2sZLD9 (my fantasy fiction book) > > > -- http://www.needhamsoftware.com (work) https://a.co/d/b2sZLD9 (my fantasy fiction book)
