In a master/slave system, it is OK to run as fast as possible to the master. In a cloud system, we want to keep the indexing load at a level that doesn’t interfere with queries.
I do this by matching the number of indexing threads to the number of CPUs. Very roughly, two threads will keep one CPU busy, that is one thread waiting for the CPU to finish the batch and another sending the next batch. With an 8 CPU machine, use 16 threads to use 100%. Or use 4 threads to use 25% (2 CPUs). In a sharded system, the indexing is spread over the leaders. For example, in our system with 8 shards, 64 threads will keep 2 CPUs busy on each leader. That number of threads runs at nearly a half-million updates per minute, so we don’t need further tuning. 2 busy CPUs is just fine on hosts with 72 CPUs. Also, we don’t use the cloud-sensitive stuff, we just throw update batches at the load balancer. One loader is a simple Python program, so that sends it all in JSON. That is the one doing 480k/min with 64 threads. Finally, we use a separate load balancer for indexing. That lets us set different response time alert levels for query traffic and update traffic. It also allows us to see anomalous bursts of query traffic separate from updates. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Mar 11, 2021, at 10:51 AM, Jan Høydahl <jan....@cominvent.com> wrote: > > Hi, > > When sending updates to Solr, you often need to run multi threaded to utilize > the CPU on the solr side. > But how can the client (whether it is pure HTTP POST or SolrJ know whether > Solr is happy with the indexing speed or not? > > I'm thinking of a feedback mechanism where Solr can check its load level, > indexing queue filling rate or other metrics as desired, and respond to the > caller with a HTTP 503, or a custom Solr HTTP code "533 Slow down". > Clients will then know that they should pause for a while and retry. Clients > can then implement an exponential backoff strategy to adjust their indexing > rate. > A bonus with such a system would be that Solr could "tell" indexing to slow > down during periods with heavy query traffic, background merge activity, > recovery, replication, if warming is too slow (max warming searchers) etc etc. > > I know Elastic has something similar. Is there already something in our APIs > that I don't know about? > > Jan
