Lots of good questions here, I’ll inline a couple of answers... > On Sep 8, 2022, at 1:59 AM, Satya Nand <satya.n...@indiamart.com.INVALID> > wrote: > > Hi All, > > We have recently moved from solr 6.5 to solr cloud 8.10. > > > *Earlier Architecture:*We were using a master-slave architecture where we > had 4 slaves(14 cpu, 96 GB ram, 20 GB Heap, 110 GB index size). We used to > optimize and replicate nightly. > > *Now.* > We didn't have a clear direction on the number of shards. So we did some > POC with variable numbers of shards. We found that with 8 shards we were > close to the response time we were getting earlier without using too much > infrastructure. > Based on our queries we couldn't find a routing parameter so now all > queries are being broadcasted to every shard. > > Now, we have 8+1 solr nodes cluster. Where 1 Indexing node contains all(8) > NRT Primary shards. This is where all indexing happens. Then We have > another 8 nodes each having ( 10 cpu, 42 GB ram,8 GB heap ~23 GB Index) > consisting of one pull replica of each primary shard. For querying, we have > used *shard.preference as PULL *so that all queries are returned from pull > replicas. > > Our thought process was that we should have the indexing layer and query > layer separate so one does not affect the other. > > we made it live this week. Though it didn't help in reducing the response > time, in fact, we found an increase in average response time. We found a > substantial impact on response time after 85 percentile response time, So > timeouts reduced significantly. > > *Now I have a few questions for all the guys who are using solr cloud to > help me understand and increase the stability of my cluster. * > > 1. Were we right to assume to separate indexing and query layer? is it a > good idea? or something else could have been done better? because right > now it can affect our cluster stability, if in case replica node is not > available then queries will start going to indexing node, which is very > weak and it could choke the whole cluster. This is a good question.. since eventually your updates go from your leaders to all your replicas, I would start with just making all nodes the same size, and not try to have two layers. I think, as you learn more, maybe you could come up with more exotic layouts, but for now, just have every node the same size, and let every node do the work. The only reason to pull out your leaders is if they somehow do EXTRA work on indexing that other nodes wouldn’t do…. > > 2. is there any guideline for the number of shards and shards size? If your queries are taking a long time due to the volume of data to query over, then shard. If you previously had a single leader/follower (the new terms for master/slave) and didn’t have performance issues, then I would have 1 shard, with 1 leader and 4 replicas. That would most closely replicate your previous setup.
> > 3. How to decide the ideal number of CPUs to have per node? is there any > metric we can follow like load or CPU usage? > what should be the ideal CPU usages and load average based on the number of > CPU ? > because our response time increases exponentially with the traffic. 250 ms > to 400 ms in peak hours. Peak hour traffic remains at 2000 requests per > minute. cpu usages at 55% and load average at ~6(10 cpu) Lots of great stuff around Grafana and other tooling to get data, but I don’t have a specific answer. > > 4. How do decide the number of nodes based on shards or any other metric? > should one increase nodes or CPUs on existing nodes? > > 5 how to handle dev and stage environments, should we have other smaller > clusters or any other approach? On dev and stage, since having more replicas is to support volume of queries, then I think you are okay with having just 1 leader and 1 follower, or even just having the leader. Now, if you need to shard to support your query cause it takes a long time, then you can do that. > > 6. Did your infrastructure requirement also increase compared to standalone > when moving to the cloud, if yes then how much? > > 7. How do you maintain versioning of config in zookeeper? I bootstrap my configset and use the api’s. https://github.com/querqy/chorus/blob/main/quickstart.sh#L119 <https://github.com/querqy/chorus/blob/main/quickstart.sh#L119> for an example. > 8, any performance issue you faced or any other recommendation? _______________________ Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
