[
https://issues.apache.org/jira/browse/SOLR-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17332818#comment-17332818
]
Walter Underwood commented on SOLR-15056:
-----------------------------------------
[~ctargett] I'm glad to fix this to follow a standard style.
The current documentation does not say which JMX metrics are used by each
circuit breaker. I added that documentation.
Some changes document what I found in the code, for example the current
documentation says:
> If circuit breakers are enabled, requests may be rejected under the condition
> of high node duress with an appropriate HTTP error code (typically 503).
>
> It is up to the client to handle this error and potentially build a retrial
> logic as this should ideally be a transient situation.
The updated doc says:
> Circuit breakers only interrupt search requests (`SearchHandler`). They are
> not checked for update requests, admin requests, etc. They are checked for
> distributed search requests, so they may result in partial failures for
> multi-shard requests. Circuit breakers are checked once, early in the request
> evaluation, before significant work is done. Long-running requests will not
> be interrupted.
>
> Servers rejecting traffic with a 503 code may mislead a load balancer into
> thinking that they are broken, when they are actually intelligently handling
> overload. This could cause Solr hosts to be dropped from a load balancer,
> causing a cascading overload on the remaining hosts. Make sure the load
> balancer is configured to allow the servers to shed excess load with 503
> responses.
The Wikipedia link is a broken URL, that is also fixed.
> CPU circuit breaker needs to use CPU utilization, not Unix load average
> -----------------------------------------------------------------------
>
> Key: SOLR-15056
> URL: https://issues.apache.org/jira/browse/SOLR-15056
> Project: Solr
> Issue Type: Bug
> Components: metrics
> Affects Versions: 8.7
> Reporter: Walter Underwood
> Assignee: Atri Sharma
> Priority: Major
> Labels: Metrics
> Attachments:
> 0001-SOLR-15056-Circuit-Breakers-use-CPU-utilization-inst.patch,
> 0002-SOLR-15056-clean-up-linkage-to-SolrCore-add-back-loa.patch,
> SOLR-15056.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The config range, 50% to 95%, assumes that the circuit breaker is triggered
> by a CPU utilization metric that goes from 0% to 100%. But the code uses the
> metric OperatingSystemMXBean.getSystemLoadAverage(). That is an average of
> the count of processes waiting to run. It is effectively unbounded. I've seen
> it as high as 50 to 100. It is not bound by 1.0 (100%).
> A good limit for load average would need to be aware of the number of CPUs
> available to the JVM. A load average of 8 is no problem for a 32 CPU host. It
> is a critical situation for a 2 CPU host.
> Also, load average is a Unix OS metric. I don't know if it is even available
> on Windows.
> Instead, use a CPU utilization metric that goes from 0.0 to 1.0. A good
> choice is OperatingSystemMXBean.getSystemCPULoad(). This name also uses
> "load", but it is a usage metric.
> From the Javadoc:
> > Returns the "recent cpu usage" for the whole system. This value is a double
> >in the [0.0,1.0] interval. A value of 0.0 means that all CPUs were idle
> >during the recent period of time observed, while a value of 1.0 means that
> >all CPUs were actively running 100% of the time during the recent period
> >being observed. All values betweens 0.0 and 1.0 are possible depending of
> >the activities going on in the system. If the system recent cpu usage is not
> >available, the method returns a negative value.
> https://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html#getSystemCpuLoad()
> Also update the documentation to explain which JMX metrics are used for the
> memory and CPU circuit breakers.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
