Yes, that is what I'm recommending customers right now, to manually match indexing threads with CPUs, and that is the "manual" way.
My question was rather whether we have or want to add some dynamic backoff system so that clients can just go full speed until told to back off, and thus adjust perfectly to what Solr can swallow. I had a client the other day where they ingested too fast, into a system with a other query load going on at the same time, and it caused some serious slowdown and even GC pauses. Jan > 11. mar. 2021 kl. 20:55 skrev Walter Underwood <wun...@wunderwood.org>: > > In a master/slave system, it is OK to run as fast as possible to the master. > In a cloud system, we want to keep the indexing load at a level that doesn’t > interfere with queries. > > I do this by matching the number of indexing threads to the number of CPUs. > Very roughly, two threads will keep one CPU busy, that is one thread waiting > for > the CPU to finish the batch and another sending the next batch. > > With an 8 CPU machine, use 16 threads to use 100%. Or use 4 threads > to use 25% (2 CPUs). > > In a sharded system, the indexing is spread over the leaders. For example, > in our system with 8 shards, 64 threads will keep 2 CPUs busy on each > leader. That number of threads runs at nearly a half-million updates per > minute, so we don’t need further tuning. 2 busy CPUs is just fine on hosts > with 72 CPUs. > > Also, we don’t use the cloud-sensitive stuff, we just throw update batches > at the load balancer. One loader is a simple Python program, so that sends > it all in JSON. That is the one doing 480k/min with 64 threads. > > Finally, we use a separate load balancer for indexing. That lets us set > different > response time alert levels for query traffic and update traffic. It also > allows us > to see anomalous bursts of query traffic separate from updates. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > >> On Mar 11, 2021, at 10:51 AM, Jan Høydahl <jan....@cominvent.com> wrote: >> >> Hi, >> >> When sending updates to Solr, you often need to run multi threaded to >> utilize the CPU on the solr side. >> But how can the client (whether it is pure HTTP POST or SolrJ know whether >> Solr is happy with the indexing speed or not? >> >> I'm thinking of a feedback mechanism where Solr can check its load level, >> indexing queue filling rate or other metrics as desired, and respond to the >> caller with a HTTP 503, or a custom Solr HTTP code "533 Slow down". >> Clients will then know that they should pause for a while and retry. Clients >> can then implement an exponential backoff strategy to adjust their indexing >> rate. >> A bonus with such a system would be that Solr could "tell" indexing to slow >> down during periods with heavy query traffic, background merge activity, >> recovery, replication, if warming is too slow (max warming searchers) etc >> etc. >> >> I know Elastic has something similar. Is there already something in our APIs >> that I don't know about? >> >> Jan >
