On 1/23/23 14:55, Dominique Bejean wrote:
Indexing rate average is 100 docs per second with autocommit each 5 minutes and autoSoftCommit each minute. Searching rate average is 1000 queries per seconds
That is a VERY high query rate. Two replicas is almost certainly not enough.
I bet you've hit a wall with performance. One of those situations where X is fine, but X+1 shows MUCH worse performance. This is a VERY common performance phenomenon with back-end processing systems including Solr.
You'll probably need to go to at least three replicas of every shard ... and even three may not be enough for the long term.
Are you relying on standard NRT replication, or have you switched to TLOG? With only two replicas you do not want to use PULL at all.
Switching to TLOG replicas *MIGHT* help, but ultimately for that high a query load you are going to need more replicas. You could try installing more memory so the index is cached better, but it's difficult to say whether it is cheaper to add memory or add servers. For servers in the cloud (like AWS) it is often cheaper to add servers than to upgrade to servers with more memory. With a billion documents, we are talking about a LOT of memory. Also, a large merge with indexes that big could take HOURS, and will impact CPU more than I/O.
For an index that big, an autoSoftCommit maxTime of one minute may be way too short. You'll want to investigate how long it takes for the soft commits to happen. The autoCommit time is probably fine, and as long as openSearcher is set to false, could probably be decreased drastically. Using opensearcher=false with autoCommit is strongly recommended, especially when also using autoSoftCommit. Usually you want your autoSoftCommit interval to be larger than your autoCommit interval.
Thanks, Shawn
