Hello all, We currently have a Solr cloud set up under version 5.4.1 running on Windows.
It contains 45 million records in a collection split across 12 shards with a replication factor of 2. The 12 shards are hosted across 6 servers running 4 nodes each. 3 servers are in one data center and 3 in another so that we can have high availability and site redundancy. This works great, serves hundreds of queries per minute and handles a trickle feed of new documents (20K a day) and updates easily, but runs on old kit. We've built a new cloud using Solr 9.1 (Eclipse Adoptium OpenJDK 17.0.4) and the same topology and settings on new servers and I have found that initial few queries on a node are really bad. P95 times across the board are 6-8K ms. It's like the first couple of queries are spinning up a new indexer before settling into memory. Once in memory (or spun up), queries return in <10ms, but leave it idle for 90 seconds and we get the slow query again. I've noticed on some occasions the admin UI panel can be slow on query if left idle. Using dev tools in the browser will also show for the first query the page time is also greater than the already high query/elapsed time. We do not autowarm caches due to the nature of the queries being used. Garbage collection looks fine on 1200mb allocation. They are quick and infrequent, usually getting to 70% of 1200mb allocated before being cleaned. We see the same behaviour without any active indexing taking place so can rule that out. The cloud was previously built on version 9 with the same issue. The only thing that helps keep queries quick is if I spam requests at the cloud constantly. I can't compare any idle periods on the 5.4 Solr cloud because there's always some activity. Perhaps once this is used in production the activity will keep things alive, but is there something else I can look at to keep the cloud active at all times? Thanks. Rydym yn croesawu derbyn gohebiaeth yng Nghymraeg. Byddwn yn ateb y fath ohebiaeth yng Nghymraeg ac ni fydd hyn yn arwain at oedi. We welcome receiving correspondence in Welsh. We will reply to such correspondence in Welsh and this will not lead to a delay.
