Hi, There are no hard rules wrt sharding, it often comes down to measuring and experimenting for your workload.
There are other things to consider than shard size. Why are the queries slow? How many rows do you ask for? Do you use faceting? Grouping? You have 25Gb of data on each of the 8 nodes/shards. Now, how much RAM does each node have, and how much RAM did you allocate to Solr/Java? A common mistake is to allocate too much ram/heap to Solr to you don't get any virtual memory caching in Linux. Say you have 32Gb of physical RAM on the nodes. Then do not give 30 of those to Solr. Instead give 8Gb to Solr and let 24Gb be available for disk caching. Other things to consider is to look at whether your queries can be optimized by rewriting them to more efficient equivalents. Sometimes, Solr-level caches can also help. Wrt shards efficiency: If you already have 8 shards, it is not much more expensive to go to 16, but you increase the risk of a single failure affecting your requests... Jan > 13. sep. 2023 kl. 10:32 skrev Saksham Gupta > <saksham.gu...@indiamart.com.INVALID>: > > Hi All, > > I have been trying to reduce the response time of solr cloud(v8.10, 8 > nodes). To achieve this, I have tried increasing the number of shards of > solr cloud which can help reduce data size on each shard thereby reducing > response time. > > > I have encountered a few questions regarding sharding strategy: > > 1. How to decide the ideal number of shards? Is there a minimum or maximum > number of shards which should be used? > > 2. What is the minimum size of a shard after which reducing the size > further won't have any effect on the response time (as time taken by other > factors like data aggregation will compensate for that) ? > > 3. Is there some maximum limit to the size of data that should be kept in a > shard? > > > As of now we have 8 shards each on a separate node with ~25 gb of > data(15-16 million docs) present on each shard. Please advise me of the > standard approaches to define the number of shards and shard size. Thanks > in advance.
