dsmiley commented on code in PR #3811:
URL: https://github.com/apache/solr/pull/3811#discussion_r2463924304
##########
solr/core/src/java/org/apache/solr/handler/admin/MetricsHandler.java:
##########
@@ -51,24 +51,15 @@
public class MetricsHandler extends RequestHandlerBase implements
PermissionNameProvider {
final SolrMetricManager metricManager;
- public static final String COMPACT_PARAM = "compact";
- public static final String PREFIX_PARAM = "prefix";
- public static final String REGEX_PARAM = "regex";
- public static final String PROPERTY_PARAM = "property";
- public static final String REGISTRY_PARAM = "registry";
- public static final String GROUP_PARAM = "group";
Review Comment:
This was a nice metrics filtering parameter; the most course filter and was
simple to understand. Did we need to throw this out? I could see continuing
to support this relatively easily. On the other hand, it might strictly
speaking be redundant. And anyone who consumed metrics certainly needs to
change what they do _anyway_. I suppose the closest thing we have going
forward is to filter by the category attribute. Still; I predict we'll add
something soon. Why internally enumerate every core registry for metrics
wanted that only exist at a core level.
##########
solr/CHANGES.txt:
##########
@@ -87,6 +87,8 @@ New Features
* SOLR-17815: Add parameter to regulate for ACORN-based filtering in vector
search. (Anna Ruggero, Alessandro Benedetti)
+* SOLR-17458: Switch from Dropwizard to OpenTelemetry. This change provides
native Prometheus support on the /admin/metrics API, OTLP support, exemplar
support for tracing correlation with OpenMetrics format and native attributes
and labels on all metrics. (Matthew Biscocho, David Smiley, Sanjay Dutt, Jude
Muriithi, Luke Kot-Zaniewski, Carlos Ugarte, Kevin Liang, Bryan Jacobowitz,
Adam Quigley)
Review Comment:
See Jan's dev-list post about the new changelog system.
New Feature category is debatable. If it's presented as a new feature;
maybe... switching from A to B doesn't sound like a new feature. Other
category is least debatable.
##########
solr/solr-ref-guide/modules/deployment-guide/pages/jmx-with-solr.adoc:
##########
Review Comment:
This removed still-valid info on connecting JMX tools to Solr (like JConsole)
##########
solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc:
##########
@@ -16,82 +16,68 @@
// specific language governing permissions and limitations
// under the License.
-Solr includes a developer API and instrumentation for the collection of
detailed performance-oriented metrics throughout the life-cycle of Solr service
and its various components.
+Solr supports both a pull-based Prometheus-formatted API and an OTLP push
exporter for collecting detailed performance-oriented metrics throughout the
lifecycle of Solr services and their various components.
-Internally this feature uses the http://metrics.dropwizard.io[Dropwizard
Metrics API], which uses the following classes of meters to measure events:
+All metrics natively include attributes and labels, providing users with
powerful ways to aggregate metrics in their preferred backend, as well as
descriptions to help understand what each metric represents.
-* *counters* - simply count events.
-They provide a single long value, e.g., the number of requests.
-* *meters* - additionally compute rates of events.
-Provide a count (as above) and 1-, 5-, and 15-minute exponentially decaying
rates, similar to the Unix system load average.
-* *histograms* - calculate approximate distribution of events according to
their values.
-Provide the following approximate statistics, with a similar exponential decay
as above: mean (arithmetic average), median, maximum, minimum, standard
deviation, and 75^th^, 95^th^, 98^th^, 99^th^ and 999^th^ percentiles.
-* *timers* - measure the number and duration of events.
-They provide a count and histogram of timings.
-* *gauges* - offer instantaneous reading of a current value, e.g., current
queue depth, current number of active connections, free heap size.
+Internally this feature uses https://opentelemetry.io/[OpenTelemetry], which
uses the following instruments to measure events:
-Some of these meters may be missing or empty for any number of valid reasons.
-In these cases, missing values of any type will be returned as `null` by
default so empty values won't impact averages or histograms.
-This is configurable for several types of missing values; see the <<The
<metrics> <missingValues> Element>> section below.
+* *Counters* - Simply count events. They only increase and never decrease,
providing a single long value (e.g., the number of requests).
+* *Gauges* - Provide instantaneous readings of current values (e.g., current
queue depth, number of active connections, free heap size).
+* *Histograms* - Provide distribution of values divided into buckets with
overall count and sum of events (e.g., duration of HTTP requests, count of
requests recorded).
-Each group of related metrics with unique names is managed in a *metric
registry*.
-Solr maintains several such registries, each corresponding to a high-level
group such as: `jvm`, `jetty`, `node`, and `core` (see <<Metric Registries>>
below).
+For more information on instruments see the OpenTelemetry documentation on
https://opentelemetry.io/docs/concepts/signals/metrics/#metric-instruments[Metric
Instruments].
Review Comment:
The list above only has 3 but OTEL's page list 7. Any way, I don't think we
need to document OTEL stuff here; we can just refer to OTEL's docs.
##########
solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc:
##########
@@ -107,771 +93,287 @@ RequestHandlers can be configured to roll up core level
metrics to the node leve
</requestHandler>
```
-=== Jetty Registry
-
-This registry is returned at `solr.jetty` and includes the following
information.
-When making requests with the <<Metrics API>>, you can specify `&group=jetty`
to limit to only these metrics.
+=== JVM Registry
-* threads and pools,
-* connection and request timers,
-* meters for responses by HTTP class (1xx, 2xx, etc.)
+The `JVM Registry` gathers metrics from the JVM using the OpenTelemetry
instrumentation library with JFR and JMX. See the
https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/runtime-telemetry/runtime-telemetry-java17/library[runtime-telemetry-java17]
documentation for more information on available JVM metrics.
-== Metrics Configuration
+JVM metrics are enabled by default but can be disabled by setting either the
system property `-Dsolr.metrics.jvm.enabled=false` or the environment variable
`SOLR_METRICS_JVM_ENABLED=false`.
-The metrics available in your system can be customized by modifying the
`<metrics>` element in `solr.xml`.
+=== Overseer Registry
-TIP: See also the section xref:configuration-guide:configuring-solr-xml.adoc[]
for more information about the `solr.xml` file, where to find it, and how to
edit it.
+The `Overseer Registry` is initialized when running in SolrCloud mode and
includes the following information:
-=== Disabling the Metrics Collection
-The `<metrics>` element in `solr.xml` supports one attribute `enabled`, which
takes a boolean value,
-for example `<metrics enabled="true">`.
+* Size of the Overseer queues (collection work queue and cluster state update
queue)
-The default value of this attribute is `true`, meaning that metrics are being
collected, processed and
-reported by Solr according to the configured metric reporters.
-They are also available from the
-metrics APIs.
+== Core Level Metrics
-The `false` value of this attribute (`<metrics enabled="false">`) turns off
metrics collection and processing.
-Internally, all metrics suppliers are replaced by singleton no-op
-implementations, which effectively removes nearly all overheads related to
metrics collection.
-All reporter configurations are skipped, and the metrics APIs stop reporting
any metrics and only return an `<error>`
-element in their responses.
+=== Index Merge Metrics
-=== The <metrics> <hiddenSysProps> Element
+These metrics are collected under the `INDEX` category and track flush
operations (documents being written to disk) and merge operations (segments on
disk being merged).
-This section of `solr.xml` allows you to define the system properties which
are considered system-sensitive and should not be exposed via the Metrics API.
+For merge metrics, metrics are tracked with the distinction of "minor" and
"major" merges (as merges with fewer documents will be typically more frequent).
+This is indicated by the `merge_type` label for the metric. The threshold for
when a merge becomes large enough to be considered major is configurable, but
+defaults to 524k documents.
-If this section is not defined, the following default configuration is used
which hides password and authentication information:
+Metrics collection for index merges can be configured in the `<metrics>`
section of `solrconfig.xml` as shown below:
[source,xml]
----
-<metrics>
- <hiddenSysProps>
- <str>javax.net.ssl.keyStorePassword</str>
- <str>javax.net.ssl.trustStorePassword</str>
- <str>solr.security.auth.basicauth.credentials</str>
- <str>zkDigestPassword</str>
- <str>zkDigestReadonlyPassword</str>
- </hiddenSysProps>
-</metrics>
+<config>
+ ...
+ <indexConfig>
+ <metrics>
+ <long name="majorMergeDocs">524288</long>
+ </metrics>
+ ...
+ </indexConfig>
+...
+</config>
----
-[#the-metrics-reporters-element]
-=== The <metrics> <reporters> Element
-
-Reporters consume the metrics data generated by Solr.
-See the section <<Reporters>> below for more details on how to configure
custom reporters.
-
-=== The <metrics> <suppliers> Element
+== Metrics API
-Suppliers help Solr generate metrics data.
-The `<metrics><suppliers>` section of `solr.xml` allows you to define your own
implementations of metrics and configure parameters for them.
+The `/admin/metrics` endpoint natively provides access to all metrics in
Prometheus format by default. You can also specify `wt=prometheus` as a
parameter for Prometheus format or `wt=openmetrics` for OpenMetrics format.
More information on the data models is provided in the sections below.
-Implementation of a custom metrics supplier is beyond the scope of this guide,
but there are other customizations possible with the default implementation,
via the elements described below.
+=== Prometheus
-`<counter>`::
-This element defines the implementation and configuration of a `Counter`
supplier.
-The default implementation does not support any configuration.
+See https://prometheus.io/docs/concepts/data_model/[Prometheus Data Model]
documentation for more information on its data model.
-`<meter>`::
-This element defines the implementation of a `Meter` supplier.
-The default implementation supports an additional parameter:
+This endpoint can be used to pull/scrape metrics to a Prometheus server or any
Prometheus-compatible backend directly from Solr.
-`<str name="clock">`:::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `user`
-|===
-+
-The type of clock to use for calculating EWMA rates.
-The supported values are:
-* `user`, which uses `System.nanoTime()`
-* `cpu`, which uses the current thread's CPU time
+==== Prometheus Setup
-`<histogram>`::
-This element defines the implementation of a `Histogram` supplier.
-This element also supports the `clock` parameter shown above with the `meter`
element, and also:
+The `prometheus-config.yml` file needs to be configured for a Prometheus
server to scrape and collect metrics. A basic configuration for SolrCloud mode
is as follows:
-`<str name="reservoir">`:::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `com.codahale.metrics.ExponentiallyDecayingReservoir`
-|===
-+
-The fully-qualified class name of the `Reservoir` implementation to use.
-The default is `com.codahale.metrics.ExponentiallyDecayingReservoir` but there
are other options available with the http://metrics.dropwizard.io/[Codahale
Metrics library] that Solr uses.
-
-`<int name="size">`:::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `1028`
-|===
-+
-The reservoir size.
-
-`<double name="alpha">`:::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `0.015`
-|===
-+
-The decay parameter.
-This is only valid for the `ExponentiallyDecayingReservoir`.
-
-`<long name="window">`:::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `300` seconds
-|===
-+
-The window size, in seconds, and only valid for the
`SlidingTimeWindowReservoir`.
-
-`<timer>`::
-This element defines an implementation of a `Timer` supplier.
-The default implementation supports the `clock` and `reservoir` parameters
described above.
-
-As an example of a section of `solr.xml` that defines some of these custom
parameters, the following defines the default `Meter` supplier with a
non-default `clock` and the default `Timer` is used with a non-default
reservoir:
-
-[source,xml]
+[source,plain]
----
-<metrics>
- <suppliers>
- <meter>
- <str name="clock">cpu</str>
- </meter>
- <timer>
- <str
name="reservoir">com.codahale.metrics.SlidingTimeWindowReservoir</str>
- <long name="window">600</long>
- </timer>
- </suppliers>
-</metrics>
+scrape_configs:
+ - job_name: 'solr'
+ metrics_path: "/solr/admin/metrics"
+ static_configs:
+ - targets: ['localhost:8983', 'localhost:7574']
----
-=== The <metrics> <missingValues> Element
-Long-lived metrics values are still reported when the underlying value is
unavailable (e.g., "INDEX.sizeInBytes" when IndexReader is closed).
-Short-lived transient metrics (such as cache entries) that are properties of
complex gauges (internally represented as `MetricsMap`) are simply skipped when
not available, and neither their names nor values appear in registries (or in
`/admin/metrics` reports).
+=== OpenMetrics
-When a missing value is encountered by default it's reported as null value,
regardless of the metrics type.
-This can be configured in the `solr.xml:/solr/metrics/missingValues` element,
which recognizes the following child elements (for string elements a JSON
payload is supported):
+OpenMetrics format is available from the `/admin/metrics` endpoint by
providing the `wt=openmetrics` parameter or by passing the Accept header
`application/openmetrics-text;version=1.0.0`. OpenMetrics is an extension of
the Prometheus format that adds additional metadata and exemplars.
-`nullNumber`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: none
-|===
-+
-The value to use when a missing (null) numeric metrics value is encountered.
+See https://prometheus.io/docs/specs/om/open_metrics_spec/[OpenMetrics Spec]
documentation for more information.
-`notANumber`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: none
-|===
-+
-The value to use when an invalid numeric value is encountered.
+OpenMetrics can be used to pull/scrape metrics to a Prometheus server or any
OpenMetrics-compatible backend directly from Solr.
-`nullString`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: none
-|===
-+
-The value to use when a missing (null) string metrics is encountered.
+==== Prometheus setup with exemplars
-`nullObject`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: none
-|===
-+
-The value to use when a missing (null) complex object is encountered.
+OpenMetrics includes `exemplars` that provide additional information and allow
users to leverage Solr's
xref:deployment-guide:distributed-tracing.adoc#distributed-tracing[OpenTelemetry
distributed tracing module] and metrics in a cohesive view for correlating
traces and metrics.
-Example configuration that returns null for missing numbers, `-1` for
-invalid numeric values, empty string for missing strings, and a Map for missing
-complex objects:
+Distributed tracing must be enabled to see exemplars. Exemplars will never
appear in OpenMetrics format otherwise. You can then scrape OpenMetrics format
to a Prometheus server or OpenMetrics-compatible backend.
-[source,xml]
-----
-<metrics>
- <missingValues>
- <null name="nullNumber"/>
- <int name="notANumber">-1</int>
- <str name="nullString"></str>
- <str name="nullObject">{"value":"missing"}</str>
- </missingValues>
-</metrics>
-----
+A basic `prometheus-config.yml` configuration for a Prometheus server in
SolrCloud mode that collects exemplars is as follows:
-=== Caching Threads Metrics ===
-The threads metrics in the JVM group can be expensive to compute, as it
requires traversing all threads.
-This can be avoided for every call to the metrics API (group=jvm) by setting a
high caching expiration interval
-(in seconds). For example, to cache the thread metrics for 5 seconds:
-
-[source,xml]
+[source,plain]
----
-<solr>
- <metrics>
- <caching>
- <int name="threadsIntervalSeconds">5</int>
- </caching>
- ...
- </metrics>
-...
-</solr>
+scrape_configs:
+ - job_name: 'solr'
+ metrics_path: "/solr/admin/metrics"
+ static_configs:
+ - targets: ['localhost:8983', 'localhost:7574']
+ params:
+ wt: ['openmetrics']
+ scrape_protocols:
+ - OpenMetricsText1.0.0
----
-== Reporters
+The Prometheus server must also be started with the command-line parameter
`--enable-feature=exemplar-storage` to collect exemplars from OpenMetrics.
-Reporter configurations are specified in `solr.xml` file in
`<metrics><reporter>` sections, for example:
+If you are using Grafana, follow the
https://grafana.com/docs/grafana/latest/fundamentals/exemplars/[Introduction to
exemplars] guide to connect your Prometheus data source and see exemplars on
Grafana panels.
-[source,xml]
-----
-<solr>
- <metrics>
- <reporter name="graphite" group="node, jvm"
class="org.apache.solr.metrics.reporters.SolrGraphiteReporter">
- <str name="host">graphite-server</str>
- <int name="port">9999</int>
- <int name="period">60</int>
- </reporter>
- <reporter name="log_metrics" group="core"
class="org.apache.solr.metrics.reporters.SolrSlf4jReporter">
- <int name="period">60</int>
- <str name="filter">QUERY./select.requestTimes</str>
- <str name="filter">QUERY./get.requestTimes</str>
- <str name="filter">UPDATE./update.requestTimes</str>
- <str name="filter">UPDATE./update.clientErrors</str>
- <str name="filter">UPDATE./update.errors</str>
- <str name="filter">SEARCHER.new.time</str>
- <str name="filter">SEARCHER.new.warmup</str>
- <str
name="logger">org.apache.solr.metrics.reporters.SolrSlf4jReporter</str>
- </reporter>
- </metrics>
-...
-</solr>
-----
+=== API Filtering
-This example configures two reporters: <<Graphite Reporter,Graphite>> and
<<SLF4J Reporter,SLF4J>>.
-See below for more details on how to configure reporters.
+A fixed set of parameters is available to filter metrics by either metric name
or base core labels. You can combine these parameters to filter only the
specific metrics you need:
-=== Reporter Arguments
-
-Reporter plugins use the following arguments:
+*NOTE: All parameters can be specified with more than one value in a request;
multiple values should be separated by a comma.*
`name`::
+
[%autowidth,frame=none]
|===
-s|Required |Default: none
-|===
-+
-The unique name of the reporter plugin.
-
-`class`::
-+
-[%autowidth,frame=none]
-|===
-s|Required |Default: none
-|===
-+
-The fully-qualified implementation class of the plugin, which must extend
`SolrMetricReporter`.
-
-`group`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: none
-|===
-+
-One or more of the predefined groups (see above).
-
-`registry`::
-+
-[%autowidth,frame=none]
-|===
|Optional |Default: none
|===
+
-One or more of valid fully-qualified registry names.
+The metric name to filter on.
-If both `group` and `registry` attributes are specified only the `group`
attribute is considered.
-If neither attribute is specified then the plugin will be used for all groups
and registries.
-Multiple group or registry names can be specified, separated by comma and/or
space.
-
-Additionally, several implementation-specific initialization arguments can be
specified in nested elements.
-There are some arguments that are common to SLF4J, Ganglia and Graphite
reporters:
-
-`period`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `60` seconds
-|===
-+
-The period in seconds between reports.
-
-`prefix`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: _empty string_
-|===
-+
-A prefix to be added to metric names, which may be helpful in logical grouping
of related Solr instances, e.g., machine name or cluster name.
-Default is empty string, i.e., just the registry name and metric name will be
used to form a fully-qualified metric name.
-
-`filter`::
+`category`::
+
[%autowidth,frame=none]
|===
|Optional |Default: none
|===
+
-If not empty then only metric names that start with this value will be
reported.
-Default is no filtering, i.e., all metrics from the selected registry will be
reported.
-
-Reporters are instantiated for every group and registry that they were
configured for, at the time when the respective components are initialized
(e.g., on JVM startup or SolrCore load).
-
-When reporters are created their configuration is validated (and e.g.,
necessary connections are established).
-Uncaught errors at this initialization stage cause the reporter to be
discarded from the running configuration.
-
-Reporters are closed when the corresponding component is being closed (e.g.,
on SolrCore close, or JVM shutdown) but metrics that they reported are still
maintained in respective registries, as explained in the previous section.
-
-The following sections provide information on implementation-specific
arguments.
-All implementation classes provided with Solr can be found under
`org.apache.solr.metrics.reporters`.
+The category label to filter on.
-=== JMX Reporter
-
-The JMX Reporter uses the `org.apache.solr.metrics.reporters.SolrJmxReporter`
class.
-
-It takes the following arguments:
-
-`domain`::
+`core`::
+
[%autowidth,frame=none]
|===
|Optional |Default: none
|===
+
-The JMX domain name.
-If not specified then the registry name will be used.
+The core name to filter on.
+More than one core can be specified in a request; multiple cores should be
separated by a comma.
-`serviceUrl`::
+`collection`::
+
[%autowidth,frame=none]
|===
|Optional |Default: none
|===
+
-The service URL for a JMX server.
-If not specified, Solr will attempt to discover if the JVM has an MBean server
and will use that address.
-See below for additional information on this.
+The collection name to filter on. This attribute is only filterable in
SolrCloud mode.
-`agentId`::
+`shard`::
+
[%autowidth,frame=none]
|===
|Optional |Default: none
|===
+
-The agent ID for a JMX server.
-Note either `serviceUrl` or `agentId` can be specified but not both.
-If both are specified then the default MBean server will be used.
-
-Object names created by this reporter are hierarchical, dot-separated but also
properly structured to form corresponding hierarchies in e.g., JConsole.
-This hierarchy consists of the following elements in the top-down order:
-
-* registry name (e.g., `solr.core.collection1.shard1.replica1`).
-Dot-separated registry names are also split into ObjectName hierarchy levels,
so that metrics for this registry will be shown under
`/solr/core/collection1/shard1/replica1` in JConsole, with each domain part
being assigned to `dom1, dom2, ... domN` property.
-* reporter name (the value of reporter's `name` attribute)
-* category, scope and name for request handlers
-* or additional `name1, name2, ... nameN` elements for metrics from other
components.
-
-The JMX Reporter replaces the JMX functionality available in Solr versions
before 7.0.
-If you have upgraded from an earlier version and have an MBean Server running
when Solr starts, Solr will automatically discover the location of the local
MBean server and use a default configuration for the SolrJmxReporter.
-
-You can start a local MBean server with a system property at startup by adding
`-Dcom.sun.management.jmxremote` to your start command.
-This will not add the reporter configuration to `solr.xml`, so if you enable
it with a system property, you must always start Solr with the system property
or JMX will not be enabled in subsequent starts.
+The shard name to filter on. This attribute is only filterable in SolrCloud
mode.
-=== SLF4J Reporter
-
-The SLF4J Reporter uses the
`org.apache.solr.metrics.reporters.SolrSlf4jReporter` class.
-
-It takes the following arguments, in addition to common arguments described
<<Reporter Arguments,above>>.
-
-`logger`::
+`replica_type`::
+
[%autowidth,frame=none]
|===
|Optional |Default: none
|===
+
-The name of the logger to use.
-Default is empty, in which case the group (or the initial part of the registry
name that identifies a metrics group) will be used if specified in the plugin
configuration.
-
-Users can specify logger name (and the corresponding logger configuration in
e.g., Log4j configuration) to output metrics-related logging to separate
file(s), which can then be processed by external applications.
-Here is an example for configuring the default `log4j2.xml` which ships in
Solr.
-This can be used in conjunction with the `solr.xml` example provided earlier
in this page to configure the SolrSlf4jReporter:
-
-[source,xml]
-----
-<Configuration>
- <Appenders>
- ...
- <RollingFile
- name="MetricsFile"
- fileName="${sys:solr.logs.dir}/solr_metrics.log"
- filePattern="${sys:solr.logs.dir}/solr_metrics.log.%i" >
- <PatternLayout>
- <Pattern>
- %d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{node_name} %X{collection}
%X{shard} %X{replica} %X{core} %X{trace_id}] %m%n
- </Pattern>
- </PatternLayout>
- <Policies>
- <OnStartupTriggeringPolicy />
- <SizeBasedTriggeringPolicy size="32 MB"/>
- </Policies>
- <DefaultRolloverStrategy max="10"/>
- </RollingFile>
- ...
- </Appenders>
+The replica type to filter on. Valid values are NRT, TLOG, or PULL. This
attribute is only filterable in SolrCloud mode.
- <Loggers>
- ...
- <Logger name="org.apache.solr.metrics.reporters.SolrSlf4jReporter"
level="info" additivity="false">
- <AppenderRef ref="MetricsFile"/>
- </Logger>
- ...
- </Loggers>
-</Configuration>
-----
+[[metrics_examples]]
+=== Examples
-Each log line produced by this reporter consists of configuration-specific
fields, and a message that follows this format:
+Request only metrics from the `foobar` collection:
[source,text]
-----
-type=COUNTER, name={}, count={}
-
-type=GAUGE, name={}, value={}
-
-type=TIMER, name={}, count={}, min={}, max={}, mean={}, stddev={}, median={},
p75={}, p95={}, p98={}, p99={}, p999={}, mean_rate={}, m1={}, m5={}, m15={},
rate_unit={}, duration_unit={}
-
-type=METER, name={}, count={}, mean_rate={}, m1={}, m5={}, m15={}, rate_unit={}
+http://localhost:8983/solr/admin/metrics?collection=foobar
-type=HISTOGRAM, name={}, count={}, min={}, max={}, mean={}, stddev={},
median={}, p75={}, p95={}, p98={}, p99={}, p999={}
-----
-
-(curly braces added here only as placeholders for actual values).
-
-Additionally, the following MDC context properties are passed to the logger
and can be used in log formats:
-
-`node_name`::
-Solr node name (for SolrCloud deployments, otherwise null), prefixed with `n:`.
-
-`registry`::
-Metric registry name, prefixed with `m:`.
-
-For reporters that are specific to a SolrCore also the following properties
are available:
-
-`collection`::
-Collection name, prefixed with `c:`.
-
-`shard`::
-Shard name, prefixed with `s:`.
-
-`replica`::
-Replica name (core node name), prefixed with `r:`.
-
-`core`::
-SolrCore name, prefixed with `x:`.
-
-`tag`::
-Reporter instance tag, prefixed with `t:`.
-
-=== Graphite Reporter
-
-The http://graphiteapp.org[Graphite] Reporter uses the
`org.apache.solr.metrics.reporters.SolrGraphiteReporter`) class.
-
-It takes the following attributes, in addition to the common attributes
<<Reporter Arguments,above>>.
-
-`host`::
-+
-[%autowidth,frame=none]
-|===
-s|Required |Default: none
-|===
-+
-The host name where Graphite server is running.
-
-`port`::
-+
-[%autowidth,frame=none]
-|===
-s|Required |Default: none
-|===
-+
-The port number for the server.
-
-`pickled`::
-+
-[%autowidth,frame=none]
-|===
-s|Required |Default: `false`
-|===
-+
-If `true`, use "pickled" Graphite protocol which may be more efficient.
-
-When plain-text protocol is used (`pickled==false`) it's possible to use this
reporter to integrate with systems other than Graphite, if they can accept
space-separated and line-oriented input over network in the following format:
+Request only the metrics with a category label of QUERY or UPDATE:
[source,text]
-----
-dot.separated.metric.name[.and.attribute] value epochTimestamp
-----
+http://localhost:8983/solr/admin/metrics?category=QUERY,UPDATE
-For example:
+Request only `solr_core_requests_total` metrics from the
`foobar_shard1_replica_n1` core:
-[source,plain]
-----
-example.solr.node.cores.loaded 1 1482932097
-example.solr.jetty.org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses.count
21 1482932097
-example.solr.jetty.org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses.m1_rate
2.5474287707930614 1482932097
-example.solr.jetty.org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses.m5_rate
3.8003171557510305 1482932097
-example.solr.jetty.org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses.m15_rate
4.0623076220244245 1482932097
-example.solr.jetty.org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses.mean_rate
0.5698031798408144 1482932097
-----
-
-== Core Level Metrics
-
-These metrics are available only on a per-core basis.
-Metrics can be aggregated across cores using Shard and Cluster reporters.
+[source,text]
+http://localhost:8983/solr/admin/metrics?name=solr_core_requests_total&core=foobar_shard1_replica_n1
-=== Index Merge Metrics
+Request only the core index size `solr_core_index_size_bytes` metrics from
collections labeled `foo` and `bar`:
-These metrics are collected under the `INDEX` category and track flush
operations (documents being written to disk) and merge operations (segments on
disk being merged).
+[source,text]
+http://localhost:8983/solr/admin/metrics?name=solr_core_index_size_bytes&collection=foo,bar
-For merge metrics, metrics are tracked with the distinction of "minor" and
"major" merges (as merges with fewer documents will be typically more frequent).
-This is indicated by the `merge_type` label for the metric. The threshold for
when a merge becomes large enough to be considered major is configurable, but
-defaults to 524k documents.
+== OTLP
-Metrics collection for index merges can be configured in the `<metrics>`
section of `solrconfig.xml` as shown below:
+For users who do not use or support pulling metrics in Prometheus format with
the `/admin/metrics` API, Solr also supports pushing metrics natively with
https://opentelemetry.io/docs/specs/otlp/[OTLP], which is a vendor-agnostic
protocol for pushing metrics via gRPC or HTTP.
-[source,xml]
-----
-<config>
- ...
- <indexConfig>
- <metrics>
- <long name="majorMergeDocs">524288</long>
- </metrics>
- ...
- </indexConfig>
-...
-</config>
-----
+OTLP is widely supported by many tools, vendors, and pipelines. See the
OpenTelemetry https://opentelemetry.io/ecosystem/vendors/[vendors list] for
more details on available and compatible options.
+=== OTLP properties
-== Metrics API
+Solr's internal OTLP exporter is disabled by default and is packaged with the
OpenTelemetry module.
-The `admin/metrics` endpoint provides access to all the metrics for all metric
groups.
+The module can be enabled with either the system property
`-Dsolr.modules=opentelemetry` or the environment variable
`SOLR_MODULES=opentelemetry`, similar to distributed tracing.
-A few query parameters are available to limit your request to only certain
metrics:
+The OTLP exporter can be configured with the supported system properties
below. These can also be set as environment variables by following these
mapping rules:
-`group`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `all`
-|===
-+
-The metric group to retrieve.
-The value `all` retrieves all metrics for all groups.
-Other possible values are: `jvm`, `jetty`, `node`, and `core`.
-More than one group can be specified in a request; multiple group names should
be separated by a comma.
+- Replace `.` with `_`
+- Convert camelCase to UPPER_SNAKE_CASE
Review Comment:
oh really? Where is this conversion happening? I don't believe EnvUtils
does this.
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc:
##########
@@ -186,11 +188,21 @@ Nowadays, the HTTP request is available via internal
APIs: `SolrQueryRequest.get
Solr upgraded to Jetty 12.x from 10.x as Jetty 10 and 11 have reached
end-of-life support. Jetty 12.x requires Java 17 or newer and is fully
compatible with Solr's new minimum requirement of Java 21. This upgrade brings
support for modern HTTP protocols and adopts the Jakarta EE 10 namespace. For
more details, see https://webtide.com/jetty-12-has-arrived/.
-=== Open Telemetry
+=== OpenTelemetry
+
+Solr 10 introduces a major overhaul by migrating from Dropwizard metrics to
OpenTelemetry (OTEL) for observability. This migration provides native
Prometheus support, OTLP support, exemplar support for tracing correlation, and
native attributes and labels on all metrics.
+
+* All metrics have been migrated to metric names instead of dot-delimited
format and now natively include attributes/labels.
Review Comment:
migrated to snake-case metric names
##########
solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc:
##########
@@ -16,82 +16,68 @@
// specific language governing permissions and limitations
// under the License.
-Solr includes a developer API and instrumentation for the collection of
detailed performance-oriented metrics throughout the life-cycle of Solr service
and its various components.
+Solr supports both a pull-based Prometheus-formatted API and an OTLP push
exporter for collecting detailed performance-oriented metrics throughout the
lifecycle of Solr services and their various components.
-Internally this feature uses the http://metrics.dropwizard.io[Dropwizard
Metrics API], which uses the following classes of meters to measure events:
+All metrics natively include attributes and labels, providing users with
powerful ways to aggregate metrics in their preferred backend, as well as
descriptions to help understand what each metric represents.
-* *counters* - simply count events.
-They provide a single long value, e.g., the number of requests.
-* *meters* - additionally compute rates of events.
-Provide a count (as above) and 1-, 5-, and 15-minute exponentially decaying
rates, similar to the Unix system load average.
-* *histograms* - calculate approximate distribution of events according to
their values.
-Provide the following approximate statistics, with a similar exponential decay
as above: mean (arithmetic average), median, maximum, minimum, standard
deviation, and 75^th^, 95^th^, 98^th^, 99^th^ and 999^th^ percentiles.
-* *timers* - measure the number and duration of events.
-They provide a count and histogram of timings.
-* *gauges* - offer instantaneous reading of a current value, e.g., current
queue depth, current number of active connections, free heap size.
+Internally this feature uses https://opentelemetry.io/[OpenTelemetry], which
uses the following instruments to measure events:
-Some of these meters may be missing or empty for any number of valid reasons.
-In these cases, missing values of any type will be returned as `null` by
default so empty values won't impact averages or histograms.
-This is configurable for several types of missing values; see the <<The
<metrics> <missingValues> Element>> section below.
+* *Counters* - Simply count events. They only increase and never decrease,
providing a single long value (e.g., the number of requests).
+* *Gauges* - Provide instantaneous readings of current values (e.g., current
queue depth, number of active connections, free heap size).
+* *Histograms* - Provide distribution of values divided into buckets with
overall count and sum of events (e.g., duration of HTTP requests, count of
requests recorded).
-Each group of related metrics with unique names is managed in a *metric
registry*.
-Solr maintains several such registries, each corresponding to a high-level
group such as: `jvm`, `jetty`, `node`, and `core` (see <<Metric Registries>>
below).
+For more information on instruments see the OpenTelemetry documentation on
https://opentelemetry.io/docs/concepts/signals/metrics/#metric-instruments[Metric
Instruments].
-For each group (and/or for each registry) there can be several *reporters*,
which are components responsible for communication of metrics from selected
registries to external systems.
-Currently implemented reporters support emitting metrics via JMX, Ganglia,
Graphite and SLF4J.
-
-There is also a dedicated `/admin/metrics` handler that can be queried to
report all or a subset of the current metrics from multiple registries.
+*NOTE: Some metrics and/or events may be missing or empty if the metric was
never recorded and therefore won't appear. Specifically, metrics will not
appear if the triggering event never occurs.*
== Metric Registries
-Solr includes multiple metric registries, which group related metrics.
+Internally, Solr categorizes metrics into registries that group related
metrics together. While users don't need to understand these registries and
should focus on the attributes attached to metrics for aggregation, being aware
of them can be helpful for knowing what metrics are available and their base
set of attributes.
-Metrics are maintained and accumulated through all lifecycles of components
from the start of the process until its shutdown - e.g., metrics for a
particular SolrCore are tracked through possibly several load, unload and/or
rename operations, and are deleted only when a core is explicitly deleted.
+Metrics are maintained and accumulated throughout all lifecycle phases of
components, from process startup until shutdown. For example, metrics for a
particular SolrCore are tracked through multiple load, unload, and rename
operations, and are only deleted when a core is explicitly deleted.
However, metrics are not persisted across process restarts; restarting Solr
will discard all collected metrics.
These are the major groups of metrics that are collected:
-=== JVM Registry
+=== Node / CoreContainer Registry
-This registry is returned at `solr.jvm` and includes the following information.
-When making requests with the <<Metrics API>>, you can specify `&group=jvm` to
limit to only these metrics.
+The `Node Registry` records metrics at the process level that are not specific
to any core. Metric names are prefixed with `solr_node_` and include the
following information:
-* direct and mapped buffer pools
-* class loading / unloading
-* OS memory, CPU time in nanoseconds, file descriptors, swap, system load
-* GC count and time
-* heap, non-heap memory and GC pools
-* number of threads, their states and deadlocks
-* System properties such as Java information, various installation directory
paths, ports, and similar information.
-You can control what appears here by modifying `solr.xml`.
+* handler requests (count, timing): collections, info, admin, configsets, etc.
+* number of cores (loaded, lazy, unloaded)
Review Comment:
there's no such thing as lazy cores anymore
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc:
##########
@@ -186,11 +188,21 @@ Nowadays, the HTTP request is available via internal
APIs: `SolrQueryRequest.get
Solr upgraded to Jetty 12.x from 10.x as Jetty 10 and 11 have reached
end-of-life support. Jetty 12.x requires Java 17 or newer and is fully
compatible with Solr's new minimum requirement of Java 21. This upgrade brings
support for modern HTTP protocols and adopts the Jakarta EE 10 namespace. For
more details, see https://webtide.com/jetty-12-has-arrived/.
-=== Open Telemetry
+=== OpenTelemetry
+
+Solr 10 introduces a major overhaul by migrating from Dropwizard metrics to
OpenTelemetry (OTEL) for observability. This migration provides native
Prometheus support, OTLP support, exemplar support for tracing correlation, and
native attributes and labels on all metrics.
+
+* All metrics have been migrated to metric names instead of dot-delimited
format and now natively include attributes/labels.
+
+* The `/admin/metrics` API now defaults to Prometheus exposition format. You
can specify `wt=prometheus` as a parameter for Prometheus format or
`wt=openmetrics` for OpenMetrics exposition format with exemplars support
(distributed tracing must be enabled to view exemplars).
-* The Prometheus exporter, JMX, SLF4J and Graphite metric reporters have been
removed. Users should migrate to using OTLP or the /admin/metrics endpoint with
external tools to get metrics to their preferred backend such as the
link:https://opentelemetry.io/docs/collector/[OTEL Collector].
+* The metrics API supports filtering by metric attributes with `name`,
`category`, `core`, `collection`, `shard`, and `replica_type` parameters.
Multiple values can be comma-separated.
Review Comment:
I wouldn't bother with this detail; you've updated the ref guide and should
link to that page.
##########
solr/solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc:
##########
Review Comment:
I suspect there are many people aware of Prometheus & Grafana, and expect
this page to exist in some form. To many users, OTEL is invisible plumbing.
What they "see" is Grafana, and to a lesser extent, Prometheus.
##########
solr/solr-ref-guide/modules/deployment-guide/pages/performance-statistics-reference.adoc:
##########
@@ -19,55 +19,55 @@
This page explains some of the statistics that Solr exposes.
There are two approaches to retrieving metrics.
-First, you can use the xref:metrics-reporting.adoc#metrics-api[Metrics API],
or you can enable JMX and get metrics from the
xref:mbean-request-handler.adoc[] or via an external tool such as JConsole.
-The below descriptions focus on retrieving the metrics using the Metrics API,
but the metric names are the same if using the MBean Request Handler or an
external tool.
+First, you can use the xref:metrics-reporting.adoc#metrics-api[Metrics API] or
push metrics with OTLP to your monitoring backend.
+The descriptions below focus on retrieving metrics using the Metrics API and
Prometheus, but the metric names are the same with OTLP.
-These statistics are per core.
-When you are running in SolrCloud mode these statistics would co-relate to the
performance of an individual replica.
+These statistics are per core. When you are running in SolrCloud mode these
statistics would co-relate to the performance of an individual replica.
+
+*Note: Solr metrics provide raw data that must be aggregated and calculated by
monitoring backends (Prometheus, Grafana, etc.). Counters can be use to
calculate rates and averages over time windows. Histograms provide raw bucket
data that backends use to calculate percentiles (p50, p75, p95, p99, p999),
averages, and other statistical measures. Solr delegates these calculations to
your monitoring system for better flexibility and reduced load on Solr.*
Review Comment:
note that asciidoc has a syntax for callout notes like this. You merely
bolded a bunch of text.
##########
solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc:
##########
@@ -16,82 +16,68 @@
// specific language governing permissions and limitations
// under the License.
-Solr includes a developer API and instrumentation for the collection of
detailed performance-oriented metrics throughout the life-cycle of Solr service
and its various components.
+Solr supports both a pull-based Prometheus-formatted API and an OTLP push
exporter for collecting detailed performance-oriented metrics throughout the
lifecycle of Solr services and their various components.
-Internally this feature uses the http://metrics.dropwizard.io[Dropwizard
Metrics API], which uses the following classes of meters to measure events:
+All metrics natively include attributes and labels, providing users with
powerful ways to aggregate metrics in their preferred backend, as well as
descriptions to help understand what each metric represents.
Review Comment:
I aren't attributes and labels the same thing?
##########
solr/solr-ref-guide/modules/deployment-guide/pages/performance-statistics-reference.adoc:
##########
@@ -19,55 +19,55 @@
This page explains some of the statistics that Solr exposes.
There are two approaches to retrieving metrics.
-First, you can use the xref:metrics-reporting.adoc#metrics-api[Metrics API],
or you can enable JMX and get metrics from the
xref:mbean-request-handler.adoc[] or via an external tool such as JConsole.
-The below descriptions focus on retrieving the metrics using the Metrics API,
but the metric names are the same if using the MBean Request Handler or an
external tool.
+First, you can use the xref:metrics-reporting.adoc#metrics-api[Metrics API] or
push metrics with OTLP to your monitoring backend.
+The descriptions below focus on retrieving metrics using the Metrics API and
Prometheus, but the metric names are the same with OTLP.
-These statistics are per core.
-When you are running in SolrCloud mode these statistics would co-relate to the
performance of an individual replica.
+These statistics are per core. When you are running in SolrCloud mode these
statistics would co-relate to the performance of an individual replica.
+
+*Note: Solr metrics provide raw data that must be aggregated and calculated by
monitoring backends (Prometheus, Grafana, etc.). Counters can be use to
calculate rates and averages over time windows. Histograms provide raw bucket
data that backends use to calculate percentiles (p50, p75, p95, p99, p999),
averages, and other statistical measures. Solr delegates these calculations to
your monitoring system for better flexibility and reduced load on Solr.*
Review Comment:
I think you might recast this note with a callout like "What about rates,
e.g. QPS?" further below. And, if you can, offer some info (or a link to such)
on doing so. Or just mention it's in Solr's official Grafana dashboard. (I
hope it is)
##########
solr/solr-ref-guide/modules/deployment-guide/pages/performance-statistics-reference.adoc:
##########
@@ -19,55 +19,55 @@
This page explains some of the statistics that Solr exposes.
There are two approaches to retrieving metrics.
-First, you can use the xref:metrics-reporting.adoc#metrics-api[Metrics API],
or you can enable JMX and get metrics from the
xref:mbean-request-handler.adoc[] or via an external tool such as JConsole.
-The below descriptions focus on retrieving the metrics using the Metrics API,
but the metric names are the same if using the MBean Request Handler or an
external tool.
+First, you can use the xref:metrics-reporting.adoc#metrics-api[Metrics API] or
push metrics with OTLP to your monitoring backend.
+The descriptions below focus on retrieving metrics using the Metrics API and
Prometheus, but the metric names are the same with OTLP.
-These statistics are per core.
-When you are running in SolrCloud mode these statistics would co-relate to the
performance of an individual replica.
+These statistics are per core. When you are running in SolrCloud mode these
statistics would co-relate to the performance of an individual replica.
+
+*Note: Solr metrics provide raw data that must be aggregated and calculated by
monitoring backends (Prometheus, Grafana, etc.). Counters can be use to
calculate rates and averages over time windows. Histograms provide raw bucket
data that backends use to calculate percentiles (p50, p75, p95, p99, p999),
averages, and other statistical measures. Solr delegates these calculations to
your monitoring system for better flexibility and reduced load on Solr.*
== Request Handler Statistics
+All handler metrics include a `handler` label that identifies the specific
handler the metric corresponds to (e.g., `/select`, `/sql`, `/export`, `/get`,
`/update`, etc.).
+This allows you to analyze statistics for individual handlers based on your
needs.
+
=== Update Request Handler
The update request handler is an endpoint to send data to Solr.
We can see how many update requests are being fired, how fast is it
performing, and other valuable information regarding requests.
-*Registry & Path:* `solr.<core>:UPDATE./update`
-
-You can request update request handler statistics with an API request such as
`\http://localhost:8983/solr/admin/metrics?group=core&prefix=UPDATE`.
Review Comment:
`group` will be missed; perhaps better named `registry`.
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc:
##########
@@ -186,11 +188,21 @@ Nowadays, the HTTP request is available via internal
APIs: `SolrQueryRequest.get
Solr upgraded to Jetty 12.x from 10.x as Jetty 10 and 11 have reached
end-of-life support. Jetty 12.x requires Java 17 or newer and is fully
compatible with Solr's new minimum requirement of Java 21. This upgrade brings
support for modern HTTP protocols and adopts the Jakarta EE 10 namespace. For
more details, see https://webtide.com/jetty-12-has-arrived/.
-=== Open Telemetry
+=== OpenTelemetry
+
+Solr 10 introduces a major overhaul by migrating from Dropwizard metrics to
OpenTelemetry (OTEL) for observability. This migration provides native
Prometheus support, OTLP support, exemplar support for tracing correlation, and
native attributes and labels on all metrics.
+
+* All metrics have been migrated to metric names instead of dot-delimited
format and now natively include attributes/labels.
+
+* The `/admin/metrics` API now defaults to Prometheus exposition format. You
can specify `wt=prometheus` as a parameter for Prometheus format or
`wt=openmetrics` for OpenMetrics exposition format with exemplars support
(distributed tracing must be enabled to view exemplars).
Review Comment:
It's important to state that it no longer returns Solr XML/JSON/javabin.
##########
solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc:
##########
@@ -107,771 +93,287 @@ RequestHandlers can be configured to roll up core level
metrics to the node leve
</requestHandler>
```
-=== Jetty Registry
-
-This registry is returned at `solr.jetty` and includes the following
information.
-When making requests with the <<Metrics API>>, you can specify `&group=jetty`
to limit to only these metrics.
+=== JVM Registry
-* threads and pools,
-* connection and request timers,
-* meters for responses by HTTP class (1xx, 2xx, etc.)
+The `JVM Registry` gathers metrics from the JVM using the OpenTelemetry
instrumentation library with JFR and JMX. See the
https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/runtime-telemetry/runtime-telemetry-java17/library[runtime-telemetry-java17]
documentation for more information on available JVM metrics.
-== Metrics Configuration
+JVM metrics are enabled by default but can be disabled by setting either the
system property `-Dsolr.metrics.jvm.enabled=false` or the environment variable
`SOLR_METRICS_JVM_ENABLED=false`.
-The metrics available in your system can be customized by modifying the
`<metrics>` element in `solr.xml`.
+=== Overseer Registry
-TIP: See also the section xref:configuration-guide:configuring-solr-xml.adoc[]
for more information about the `solr.xml` file, where to find it, and how to
edit it.
+The `Overseer Registry` is initialized when running in SolrCloud mode and
includes the following information:
-=== Disabling the Metrics Collection
-The `<metrics>` element in `solr.xml` supports one attribute `enabled`, which
takes a boolean value,
-for example `<metrics enabled="true">`.
+* Size of the Overseer queues (collection work queue and cluster state update
queue)
-The default value of this attribute is `true`, meaning that metrics are being
collected, processed and
-reported by Solr according to the configured metric reporters.
-They are also available from the
-metrics APIs.
+== Core Level Metrics
-The `false` value of this attribute (`<metrics enabled="false">`) turns off
metrics collection and processing.
-Internally, all metrics suppliers are replaced by singleton no-op
-implementations, which effectively removes nearly all overheads related to
metrics collection.
-All reporter configurations are skipped, and the metrics APIs stop reporting
any metrics and only return an `<error>`
-element in their responses.
+=== Index Merge Metrics
-=== The <metrics> <hiddenSysProps> Element
+These metrics are collected under the `INDEX` category and track flush
operations (documents being written to disk) and merge operations (segments on
disk being merged).
-This section of `solr.xml` allows you to define the system properties which
are considered system-sensitive and should not be exposed via the Metrics API.
+For merge metrics, metrics are tracked with the distinction of "minor" and
"major" merges (as merges with fewer documents will be typically more frequent).
+This is indicated by the `merge_type` label for the metric. The threshold for
when a merge becomes large enough to be considered major is configurable, but
+defaults to 524k documents.
-If this section is not defined, the following default configuration is used
which hides password and authentication information:
+Metrics collection for index merges can be configured in the `<metrics>`
section of `solrconfig.xml` as shown below:
[source,xml]
----
-<metrics>
- <hiddenSysProps>
- <str>javax.net.ssl.keyStorePassword</str>
- <str>javax.net.ssl.trustStorePassword</str>
- <str>solr.security.auth.basicauth.credentials</str>
- <str>zkDigestPassword</str>
- <str>zkDigestReadonlyPassword</str>
- </hiddenSysProps>
-</metrics>
+<config>
+ ...
+ <indexConfig>
+ <metrics>
+ <long name="majorMergeDocs">524288</long>
+ </metrics>
+ ...
+ </indexConfig>
+...
+</config>
----
-[#the-metrics-reporters-element]
-=== The <metrics> <reporters> Element
-
-Reporters consume the metrics data generated by Solr.
-See the section <<Reporters>> below for more details on how to configure
custom reporters.
-
-=== The <metrics> <suppliers> Element
+== Metrics API
-Suppliers help Solr generate metrics data.
-The `<metrics><suppliers>` section of `solr.xml` allows you to define your own
implementations of metrics and configure parameters for them.
+The `/admin/metrics` endpoint natively provides access to all metrics in
Prometheus format by default. You can also specify `wt=prometheus` as a
parameter for Prometheus format or `wt=openmetrics` for OpenMetrics format.
More information on the data models is provided in the sections below.
-Implementation of a custom metrics supplier is beyond the scope of this guide,
but there are other customizations possible with the default implementation,
via the elements described below.
+=== Prometheus
-`<counter>`::
-This element defines the implementation and configuration of a `Counter`
supplier.
-The default implementation does not support any configuration.
+See https://prometheus.io/docs/concepts/data_model/[Prometheus Data Model]
documentation for more information on its data model.
-`<meter>`::
-This element defines the implementation of a `Meter` supplier.
-The default implementation supports an additional parameter:
+This endpoint can be used to pull/scrape metrics to a Prometheus server or any
Prometheus-compatible backend directly from Solr.
-`<str name="clock">`:::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `user`
-|===
-+
-The type of clock to use for calculating EWMA rates.
-The supported values are:
-* `user`, which uses `System.nanoTime()`
-* `cpu`, which uses the current thread's CPU time
+==== Prometheus Setup
-`<histogram>`::
-This element defines the implementation of a `Histogram` supplier.
-This element also supports the `clock` parameter shown above with the `meter`
element, and also:
+The `prometheus-config.yml` file needs to be configured for a Prometheus
server to scrape and collect metrics. A basic configuration for SolrCloud mode
is as follows:
-`<str name="reservoir">`:::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `com.codahale.metrics.ExponentiallyDecayingReservoir`
-|===
-+
-The fully-qualified class name of the `Reservoir` implementation to use.
-The default is `com.codahale.metrics.ExponentiallyDecayingReservoir` but there
are other options available with the http://metrics.dropwizard.io/[Codahale
Metrics library] that Solr uses.
-
-`<int name="size">`:::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `1028`
-|===
-+
-The reservoir size.
-
-`<double name="alpha">`:::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `0.015`
-|===
-+
-The decay parameter.
-This is only valid for the `ExponentiallyDecayingReservoir`.
-
-`<long name="window">`:::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `300` seconds
-|===
-+
-The window size, in seconds, and only valid for the
`SlidingTimeWindowReservoir`.
-
-`<timer>`::
-This element defines an implementation of a `Timer` supplier.
-The default implementation supports the `clock` and `reservoir` parameters
described above.
-
-As an example of a section of `solr.xml` that defines some of these custom
parameters, the following defines the default `Meter` supplier with a
non-default `clock` and the default `Timer` is used with a non-default
reservoir:
-
-[source,xml]
+[source,plain]
----
-<metrics>
- <suppliers>
- <meter>
- <str name="clock">cpu</str>
- </meter>
- <timer>
- <str
name="reservoir">com.codahale.metrics.SlidingTimeWindowReservoir</str>
- <long name="window">600</long>
- </timer>
- </suppliers>
-</metrics>
+scrape_configs:
+ - job_name: 'solr'
+ metrics_path: "/solr/admin/metrics"
+ static_configs:
+ - targets: ['localhost:8983', 'localhost:7574']
----
-=== The <metrics> <missingValues> Element
-Long-lived metrics values are still reported when the underlying value is
unavailable (e.g., "INDEX.sizeInBytes" when IndexReader is closed).
-Short-lived transient metrics (such as cache entries) that are properties of
complex gauges (internally represented as `MetricsMap`) are simply skipped when
not available, and neither their names nor values appear in registries (or in
`/admin/metrics` reports).
+=== OpenMetrics
-When a missing value is encountered by default it's reported as null value,
regardless of the metrics type.
-This can be configured in the `solr.xml:/solr/metrics/missingValues` element,
which recognizes the following child elements (for string elements a JSON
payload is supported):
+OpenMetrics format is available from the `/admin/metrics` endpoint by
providing the `wt=openmetrics` parameter or by passing the Accept header
`application/openmetrics-text;version=1.0.0`. OpenMetrics is an extension of
the Prometheus format that adds additional metadata and exemplars.
-`nullNumber`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: none
-|===
-+
-The value to use when a missing (null) numeric metrics value is encountered.
+See https://prometheus.io/docs/specs/om/open_metrics_spec/[OpenMetrics Spec]
documentation for more information.
-`notANumber`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: none
-|===
-+
-The value to use when an invalid numeric value is encountered.
+OpenMetrics can be used to pull/scrape metrics to a Prometheus server or any
OpenMetrics-compatible backend directly from Solr.
-`nullString`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: none
-|===
-+
-The value to use when a missing (null) string metrics is encountered.
+==== Prometheus setup with exemplars
-`nullObject`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: none
-|===
-+
-The value to use when a missing (null) complex object is encountered.
+OpenMetrics includes `exemplars` that provide additional information and allow
users to leverage Solr's
xref:deployment-guide:distributed-tracing.adoc#distributed-tracing[OpenTelemetry
distributed tracing module] and metrics in a cohesive view for correlating
traces and metrics.
-Example configuration that returns null for missing numbers, `-1` for
-invalid numeric values, empty string for missing strings, and a Map for missing
-complex objects:
+Distributed tracing must be enabled to see exemplars. Exemplars will never
appear in OpenMetrics format otherwise. You can then scrape OpenMetrics format
to a Prometheus server or OpenMetrics-compatible backend.
-[source,xml]
-----
-<metrics>
- <missingValues>
- <null name="nullNumber"/>
- <int name="notANumber">-1</int>
- <str name="nullString"></str>
- <str name="nullObject">{"value":"missing"}</str>
- </missingValues>
-</metrics>
-----
+A basic `prometheus-config.yml` configuration for a Prometheus server in
SolrCloud mode that collects exemplars is as follows:
-=== Caching Threads Metrics ===
-The threads metrics in the JVM group can be expensive to compute, as it
requires traversing all threads.
-This can be avoided for every call to the metrics API (group=jvm) by setting a
high caching expiration interval
-(in seconds). For example, to cache the thread metrics for 5 seconds:
-
-[source,xml]
+[source,plain]
----
-<solr>
- <metrics>
- <caching>
- <int name="threadsIntervalSeconds">5</int>
- </caching>
- ...
- </metrics>
-...
-</solr>
+scrape_configs:
+ - job_name: 'solr'
+ metrics_path: "/solr/admin/metrics"
+ static_configs:
+ - targets: ['localhost:8983', 'localhost:7574']
+ params:
+ wt: ['openmetrics']
+ scrape_protocols:
+ - OpenMetricsText1.0.0
----
-== Reporters
+The Prometheus server must also be started with the command-line parameter
`--enable-feature=exemplar-storage` to collect exemplars from OpenMetrics.
-Reporter configurations are specified in `solr.xml` file in
`<metrics><reporter>` sections, for example:
+If you are using Grafana, follow the
https://grafana.com/docs/grafana/latest/fundamentals/exemplars/[Introduction to
exemplars] guide to connect your Prometheus data source and see exemplars on
Grafana panels.
-[source,xml]
-----
-<solr>
- <metrics>
- <reporter name="graphite" group="node, jvm"
class="org.apache.solr.metrics.reporters.SolrGraphiteReporter">
- <str name="host">graphite-server</str>
- <int name="port">9999</int>
- <int name="period">60</int>
- </reporter>
- <reporter name="log_metrics" group="core"
class="org.apache.solr.metrics.reporters.SolrSlf4jReporter">
- <int name="period">60</int>
- <str name="filter">QUERY./select.requestTimes</str>
- <str name="filter">QUERY./get.requestTimes</str>
- <str name="filter">UPDATE./update.requestTimes</str>
- <str name="filter">UPDATE./update.clientErrors</str>
- <str name="filter">UPDATE./update.errors</str>
- <str name="filter">SEARCHER.new.time</str>
- <str name="filter">SEARCHER.new.warmup</str>
- <str
name="logger">org.apache.solr.metrics.reporters.SolrSlf4jReporter</str>
- </reporter>
- </metrics>
-...
-</solr>
-----
+=== API Filtering
-This example configures two reporters: <<Graphite Reporter,Graphite>> and
<<SLF4J Reporter,SLF4J>>.
-See below for more details on how to configure reporters.
+A fixed set of parameters is available to filter metrics by either metric name
or base core labels. You can combine these parameters to filter only the
specific metrics you need:
-=== Reporter Arguments
-
-Reporter plugins use the following arguments:
+*NOTE: All parameters can be specified with more than one value in a request;
multiple values should be separated by a comma.*
`name`::
+
[%autowidth,frame=none]
|===
-s|Required |Default: none
-|===
-+
-The unique name of the reporter plugin.
-
-`class`::
-+
-[%autowidth,frame=none]
-|===
-s|Required |Default: none
-|===
-+
-The fully-qualified implementation class of the plugin, which must extend
`SolrMetricReporter`.
-
-`group`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: none
-|===
-+
-One or more of the predefined groups (see above).
-
-`registry`::
-+
-[%autowidth,frame=none]
-|===
|Optional |Default: none
|===
+
-One or more of valid fully-qualified registry names.
+The metric name to filter on.
-If both `group` and `registry` attributes are specified only the `group`
attribute is considered.
-If neither attribute is specified then the plugin will be used for all groups
and registries.
-Multiple group or registry names can be specified, separated by comma and/or
space.
-
-Additionally, several implementation-specific initialization arguments can be
specified in nested elements.
-There are some arguments that are common to SLF4J, Ganglia and Graphite
reporters:
-
-`period`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: `60` seconds
-|===
-+
-The period in seconds between reports.
-
-`prefix`::
-+
-[%autowidth,frame=none]
-|===
-|Optional |Default: _empty string_
-|===
-+
-A prefix to be added to metric names, which may be helpful in logical grouping
of related Solr instances, e.g., machine name or cluster name.
-Default is empty string, i.e., just the registry name and metric name will be
used to form a fully-qualified metric name.
-
-`filter`::
+`category`::
+
[%autowidth,frame=none]
|===
|Optional |Default: none
|===
+
-If not empty then only metric names that start with this value will be
reported.
-Default is no filtering, i.e., all metrics from the selected registry will be
reported.
-
-Reporters are instantiated for every group and registry that they were
configured for, at the time when the respective components are initialized
(e.g., on JVM startup or SolrCore load).
-
-When reporters are created their configuration is validated (and e.g.,
necessary connections are established).
-Uncaught errors at this initialization stage cause the reporter to be
discarded from the running configuration.
-
-Reporters are closed when the corresponding component is being closed (e.g.,
on SolrCore close, or JVM shutdown) but metrics that they reported are still
maintained in respective registries, as explained in the previous section.
-
-The following sections provide information on implementation-specific
arguments.
-All implementation classes provided with Solr can be found under
`org.apache.solr.metrics.reporters`.
+The category label to filter on.
-=== JMX Reporter
-
-The JMX Reporter uses the `org.apache.solr.metrics.reporters.SolrJmxReporter`
class.
-
-It takes the following arguments:
-
-`domain`::
+`core`::
+
[%autowidth,frame=none]
|===
|Optional |Default: none
|===
+
-The JMX domain name.
-If not specified then the registry name will be used.
+The core name to filter on.
+More than one core can be specified in a request; multiple cores should be
separated by a comma.
-`serviceUrl`::
+`collection`::
+
[%autowidth,frame=none]
|===
|Optional |Default: none
|===
+
-The service URL for a JMX server.
-If not specified, Solr will attempt to discover if the JVM has an MBean server
and will use that address.
-See below for additional information on this.
+The collection name to filter on. This attribute is only filterable in
SolrCloud mode.
-`agentId`::
+`shard`::
+
[%autowidth,frame=none]
|===
|Optional |Default: none
|===
+
-The agent ID for a JMX server.
-Note either `serviceUrl` or `agentId` can be specified but not both.
-If both are specified then the default MBean server will be used.
-
-Object names created by this reporter are hierarchical, dot-separated but also
properly structured to form corresponding hierarchies in e.g., JConsole.
-This hierarchy consists of the following elements in the top-down order:
-
-* registry name (e.g., `solr.core.collection1.shard1.replica1`).
-Dot-separated registry names are also split into ObjectName hierarchy levels,
so that metrics for this registry will be shown under
`/solr/core/collection1/shard1/replica1` in JConsole, with each domain part
being assigned to `dom1, dom2, ... domN` property.
-* reporter name (the value of reporter's `name` attribute)
-* category, scope and name for request handlers
-* or additional `name1, name2, ... nameN` elements for metrics from other
components.
-
-The JMX Reporter replaces the JMX functionality available in Solr versions
before 7.0.
-If you have upgraded from an earlier version and have an MBean Server running
when Solr starts, Solr will automatically discover the location of the local
MBean server and use a default configuration for the SolrJmxReporter.
-
-You can start a local MBean server with a system property at startup by adding
`-Dcom.sun.management.jmxremote` to your start command.
-This will not add the reporter configuration to `solr.xml`, so if you enable
it with a system property, you must always start Solr with the system property
or JMX will not be enabled in subsequent starts.
+The shard name to filter on. This attribute is only filterable in SolrCloud
mode.
-=== SLF4J Reporter
-
-The SLF4J Reporter uses the
`org.apache.solr.metrics.reporters.SolrSlf4jReporter` class.
-
-It takes the following arguments, in addition to common arguments described
<<Reporter Arguments,above>>.
-
-`logger`::
+`replica_type`::
+
[%autowidth,frame=none]
|===
|Optional |Default: none
|===
+
-The name of the logger to use.
-Default is empty, in which case the group (or the initial part of the registry
name that identifies a metrics group) will be used if specified in the plugin
configuration.
-
-Users can specify logger name (and the corresponding logger configuration in
e.g., Log4j configuration) to output metrics-related logging to separate
file(s), which can then be processed by external applications.
-Here is an example for configuring the default `log4j2.xml` which ships in
Solr.
-This can be used in conjunction with the `solr.xml` example provided earlier
in this page to configure the SolrSlf4jReporter:
-
-[source,xml]
-----
-<Configuration>
- <Appenders>
- ...
- <RollingFile
- name="MetricsFile"
- fileName="${sys:solr.logs.dir}/solr_metrics.log"
- filePattern="${sys:solr.logs.dir}/solr_metrics.log.%i" >
- <PatternLayout>
- <Pattern>
- %d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{node_name} %X{collection}
%X{shard} %X{replica} %X{core} %X{trace_id}] %m%n
- </Pattern>
- </PatternLayout>
- <Policies>
- <OnStartupTriggeringPolicy />
- <SizeBasedTriggeringPolicy size="32 MB"/>
- </Policies>
- <DefaultRolloverStrategy max="10"/>
- </RollingFile>
- ...
- </Appenders>
+The replica type to filter on. Valid values are NRT, TLOG, or PULL. This
attribute is only filterable in SolrCloud mode.
- <Loggers>
- ...
- <Logger name="org.apache.solr.metrics.reporters.SolrSlf4jReporter"
level="info" additivity="false">
- <AppenderRef ref="MetricsFile"/>
- </Logger>
- ...
- </Loggers>
-</Configuration>
-----
+[[metrics_examples]]
+=== Examples
-Each log line produced by this reporter consists of configuration-specific
fields, and a message that follows this format:
+Request only metrics from the `foobar` collection:
[source,text]
-----
-type=COUNTER, name={}, count={}
-
-type=GAUGE, name={}, value={}
-
-type=TIMER, name={}, count={}, min={}, max={}, mean={}, stddev={}, median={},
p75={}, p95={}, p98={}, p99={}, p999={}, mean_rate={}, m1={}, m5={}, m15={},
rate_unit={}, duration_unit={}
-
-type=METER, name={}, count={}, mean_rate={}, m1={}, m5={}, m15={}, rate_unit={}
+http://localhost:8983/solr/admin/metrics?collection=foobar
-type=HISTOGRAM, name={}, count={}, min={}, max={}, mean={}, stddev={},
median={}, p75={}, p95={}, p98={}, p99={}, p999={}
-----
-
-(curly braces added here only as placeholders for actual values).
-
-Additionally, the following MDC context properties are passed to the logger
and can be used in log formats:
-
-`node_name`::
-Solr node name (for SolrCloud deployments, otherwise null), prefixed with `n:`.
-
-`registry`::
-Metric registry name, prefixed with `m:`.
-
-For reporters that are specific to a SolrCore also the following properties
are available:
-
-`collection`::
-Collection name, prefixed with `c:`.
-
-`shard`::
-Shard name, prefixed with `s:`.
-
-`replica`::
-Replica name (core node name), prefixed with `r:`.
-
-`core`::
-SolrCore name, prefixed with `x:`.
-
-`tag`::
-Reporter instance tag, prefixed with `t:`.
-
-=== Graphite Reporter
-
-The http://graphiteapp.org[Graphite] Reporter uses the
`org.apache.solr.metrics.reporters.SolrGraphiteReporter`) class.
-
-It takes the following attributes, in addition to the common attributes
<<Reporter Arguments,above>>.
-
-`host`::
-+
-[%autowidth,frame=none]
-|===
-s|Required |Default: none
-|===
-+
-The host name where Graphite server is running.
-
-`port`::
-+
-[%autowidth,frame=none]
-|===
-s|Required |Default: none
-|===
-+
-The port number for the server.
-
-`pickled`::
-+
-[%autowidth,frame=none]
-|===
-s|Required |Default: `false`
-|===
-+
-If `true`, use "pickled" Graphite protocol which may be more efficient.
-
-When plain-text protocol is used (`pickled==false`) it's possible to use this
reporter to integrate with systems other than Graphite, if they can accept
space-separated and line-oriented input over network in the following format:
+Request only the metrics with a category label of QUERY or UPDATE:
[source,text]
-----
-dot.separated.metric.name[.and.attribute] value epochTimestamp
-----
+http://localhost:8983/solr/admin/metrics?category=QUERY,UPDATE
-For example:
+Request only `solr_core_requests_total` metrics from the
`foobar_shard1_replica_n1` core:
-[source,plain]
-----
-example.solr.node.cores.loaded 1 1482932097
-example.solr.jetty.org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses.count
21 1482932097
-example.solr.jetty.org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses.m1_rate
2.5474287707930614 1482932097
-example.solr.jetty.org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses.m5_rate
3.8003171557510305 1482932097
-example.solr.jetty.org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses.m15_rate
4.0623076220244245 1482932097
-example.solr.jetty.org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses.mean_rate
0.5698031798408144 1482932097
-----
-
-== Core Level Metrics
-
-These metrics are available only on a per-core basis.
-Metrics can be aggregated across cores using Shard and Cluster reporters.
+[source,text]
+http://localhost:8983/solr/admin/metrics?name=solr_core_requests_total&core=foobar_shard1_replica_n1
-=== Index Merge Metrics
+Request only the core index size `solr_core_index_size_bytes` metrics from
collections labeled `foo` and `bar`:
-These metrics are collected under the `INDEX` category and track flush
operations (documents being written to disk) and merge operations (segments on
disk being merged).
+[source,text]
+http://localhost:8983/solr/admin/metrics?name=solr_core_index_size_bytes&collection=foo,bar
-For merge metrics, metrics are tracked with the distinction of "minor" and
"major" merges (as merges with fewer documents will be typically more frequent).
-This is indicated by the `merge_type` label for the metric. The threshold for
when a merge becomes large enough to be considered major is configurable, but
-defaults to 524k documents.
+== OTLP
-Metrics collection for index merges can be configured in the `<metrics>`
section of `solrconfig.xml` as shown below:
+For users who do not use or support pulling metrics in Prometheus format with
the `/admin/metrics` API, Solr also supports pushing metrics natively with
https://opentelemetry.io/docs/specs/otlp/[OTLP], which is a vendor-agnostic
protocol for pushing metrics via gRPC or HTTP.
-[source,xml]
-----
-<config>
- ...
- <indexConfig>
- <metrics>
- <long name="majorMergeDocs">524288</long>
- </metrics>
- ...
- </indexConfig>
-...
-</config>
-----
+OTLP is widely supported by many tools, vendors, and pipelines. See the
OpenTelemetry https://opentelemetry.io/ecosystem/vendors/[vendors list] for
more details on available and compatible options.
+=== OTLP properties
-== Metrics API
+Solr's internal OTLP exporter is disabled by default and is packaged with the
OpenTelemetry module.
Review Comment:
does the internal exporter support filtering which metrics should be
exported? I suspect it may be a problem to send *everything*
##########
solr/solr-ref-guide/modules/deployment-guide/pages/performance-statistics-reference.adoc:
##########
@@ -76,149 +76,82 @@ Other types of data such as errors and timeouts are also
provided.
These are available under different metric names.
For example:
-*
`\http://localhost:8983/solr/admin/metrics?group=core&prefix=UPDATE./update.errors`
-*
`\http://localhost:8983/solr/admin/metrics?group=core&prefix=QUERY./select.errors`
+*
`\http://localhost:8983/solr/admin/metrics?name=solr_core_requests_errors_total`
+*
`\http://localhost:8983/solr/admin/metrics?name=solr_core_requests_timeout_total`
The table below shows the metric names and attributes to request:
[cols="30,70",options="header"]
|===
|Metric name | Description
-|`QUERY./select.errors`
-`UPDATE./update.errors` |Number of errors encountered by handler. In addition
to a count of errors, mean, 1 minute, 5 minute, and 15 minute rates are also
available.
-|`QUERY./select.clientErrors`
-`UPDATE./update.clientErrors` |Number of syntax or parse errors made by a
client while making requests. In addition to a count of errors, mean, 1 minute,
5 minute, and 15 minute rates are also available.
-|`QUERY./select.requests`
-`UPDATE./update.requests` |Number of requests received by this handler.
-|`QUERY./select.serverErrors`
-`UPDATE./update.serverErrors` |Number of errors thrown by the server while
executing the request. In addition to a count of errors, mean, 1 minute, 5
minute, and 15 minute rates are also available.
-|`QUERY./select.timeouts`
-`UPDATE./update.timeouts` |Number of responses received with partial results.
In addition to a count of timeout events, mean, 1 minute, 5 minute, and 15
minute rates are also available.
-|`QUERY./select.totalTime`
-`UPDATE./update.totalTime` |The sum of all request processing times since the
Solr process was started in nanoseconds.
-|`QUERY./select.handlerStart`
-`UPDATE./update.handlerStart` |Epoch time when the handler was registered.
+|`solr_core_requests_errors_total` |Number of errors encountered by handler.
Uses `source` attribute to differentiate between client errors (bad requests,
malformed queries) and server errors (internal failures). Client errors have
`source="client"` while server errors have `source="server"`.
+|`solr_core_requests_timeout_total` |Counter for requests that were cancelled
due to timeouts.
|===
*Differentiating Internal Requests*
Processing of a single request in SolrCloud for a large collection requires
making additional requests to other replicas, often on other nodes.
The internal requests look much the same on the surface (same handler), but
they are performing a portion of the over-arching task.
Differentiating these requests is really important!
-Solr tracks its metrics on these handlers with a different handler name when
the request is contributing to some other request:
+Solr tracks its metrics on these handlers with an `internal` attribute when
the request is contributing to some other request:
-* Queries: `/select` query's internal requests will be tracked as
`/select[shard]`. Technically, this occurs on `SearchHandler` and its
subclasses.
+* Queries: `/select` query's internal requests will have `internal=true` in
their metric attributes. This occurs on `SearchHandler` and its subclasses.
* _(More can be instrumented some day)_
-Solr's Prometheus exporter configuration extracts this suffix on the handler
to a label named "internal". When configuring Grafana or other metrics tools,
be sure to filter these metrics in or out depending on what is being analyzed.
+When using external monitoring tools like Prometheus or Grafana, be sure to
filter metrics based on the `internal` attribute depending on what is being
analyzed.
== Update Handler
This section has information on the total number of adds and how many commits
have been fired against a Solr core.
-*Registry & Path:* `solr.<core>:UPDATE.updateHandler`
-
-You can get all update handler statistics shown in the table below with an API
request such as
`\http://localhost:8983/solr/admin/metrics?group=core&prefix=UPDATE.updateHandler`.
+You can get all update handler statistics shown in the table below with an API
request such as
`\http://localhost:8983/solr/admin/metrics?category=UPDATE,TLOG`.
The following describes the specific statistics you can get:
[%autowidth.stretch,options="header"]
|===
-|Attribute |Description
-|`UPDATE.updateHandler.adds` |Total number of “add” requests since last commit.
-|`UPDATE.updateHandler.autoCommitMaxTime` |Maximum time between two
auto-commits execution.
-|`UPDATE.updateHandler.autoCommits` |Total number of auto-commits executed.
-|`UPDATE.updateHandler.commits` | Number of total commits executed.
-
-In addition to a count of commits, mean, 1 minute, 5 minute, and 15 minute
rates are also available.
-|`UPDATE.updateHandler.cumulativeAdds` |Number of “effective” additions
executed over the lifetime. The counter is incremented when “add‘ command is
executed while decremented when “rollback” is executed.
-
-In addition to a count of adds, mean, 1 minute, 5 minute, and 15 minute rates
are also available.
-|`UPDATE.updateHandler.cumulativeDeletesById` |Number of document deletions
executed by ID over the lifetime. The counter is incremented when “delete”
command is executed and decremented when “rollback” is executed.
-
-In addition to a count of deletes, mean, 1 minute, 5 minute, and 15 minute
rates are also available.
-|`UPDATE.updateHandler.cumulativeDeletesByQuery` |Number of document deletions
executed by query over the lifetime. The counter is incremented when “delete”
command is executed and decremented when “rollback” is executed.
-
-In addition to a count of deletes, mean, 1 minute, 5 minute, and 15 minute
rates are also available.
-|`UPDATE.updateHandler.cumulativeErrors` |Number of error messages received
while performing addition/deletion actions on documents over the lifetime.
-
-In addition to a count of errors, mean, 1 minute, 5 minute, and 15 minute
rates are also available.
-|`UPDATE.updateHandler.deletesById` |Currently uncommitted deletions by ID.
-|`UPDATE.updateHandler.deletesByQuery` |Currently uncommitted deletions by
query.
-|`UPDATE.updateHandler.docsPending` |Number of documents which are pending
commit.
-|`UPDATE.updateHandler.errors` |Number of error messages received while
performing addition/deletion/commit/rollback actions on documents over the
lifetime of the core.
-|`UPDATE.updateHandler.expungeDeletes` |Number of commit commands issued with
expunge deletes.
+|Metric |Description
+|`solr_core_update_submitted_ops` |Counter for operations submitted to the
update handler.
+|`solr_core_update_committed_ops` |Counter for operations that have been
committed.
+|`solr_core_update_cumulative_ops` |Gauge showing cumulative count of
operations over the lifetime. Cumulative can decrease from rollback command.
+|`solr_core_update_commit_ops` |Counter for commit operations.
+|`solr_core_update_maintenance_ops` |Counter for total number of maintenance
operations such as rollback
+|`solr_core_update_docs_pending_commit` |Gauge showing number of documents
pending commit.
+|`solr_core_update_log_buffered_ops` |Gauge for current number of buffered
operations.
+|`solr_core_update_log_replay_logs_remaining` |Gauge current number of tlogs
remaining to be replayed.
+|`solr_core_update_log_size_remaining` |Gauge total size in bytes of all tlogs
remaining to be replayed.
+|`solr_core_update_log_state` |Gauge The current state of the update log.
Replaying (0), buffering (1), applying buffered (2), active (3).
+|`solr_core_update_log_applied_buffered_ops` |Counter number of buffered
operations applied.
+|===
-In addition to a count of expunge deletes, mean, 1 minute, 5 minute, and 15
minute rates are also available.
-|`UPDATE.updateHandler.merges` | Number of index merges that have occurred.
+== Cache Statistics
-In addition to a count of merges, mean, 1 minute, 5 minute, and 15 minute
rates are also available.
-|`UPDATE.updateHandler.optimizes` |Number of explicit optimize commands issued.
+You can get the statistics shown in the table below with an API request such
as `\http://localhost:8983/solr/admin/metrics?category=CACHE`. Each cache
metric has a `name` attribute attached that correspond to the cache the metric
was recorded from.
-In addition to a count of optimizations, mean, 1 minute, 5 minute, and 15
minute rates are also available.
-|`UPDATE.updateHandler.rollbacks` |Number of rollbacks executed.
+The following statistics are available for each of the caches mentioned below:
-In addition to a count of rollbacks, mean, 1 minute, 5 minute, and 15 minute
rates are also available.
-|`UPDATE.updateHandler.softAutoCommitMaxTime` |Maximum document ‘adds’ between
two soft auto-commits.
-|`UPDATE.updateHandler.softAutoCommits` |Number of soft commits executed.
+[%autowidth.stretch,options="header"]
+|===
+|Metric Name |Description
+|`solr_caffeine_cache_ops` |Number of cumulative cache operations (inserts and
evictions).
+|`solr_caffeine_cache_lookups` |Number of cumulative cache lookup results
(hits and misses).
+|`solr_caffeine_cache_size` |Current number of cache entries.
+|`solr_caffeine_cache_ram_used` |RAM bytes used by cache.
|===
-
-== Cache Statistics
=== Document Cache
This cache holds Lucene Document objects (the stored fields for each document).
Since Lucene internal document IDs are transient, this cache cannot be
auto-warmed.
-*Registry and Path:* `solr.<core>:CACHE.searcher.documentCache`
-
-You can get the statistics shown in the table below with an API request such
as
`\http://localhost:8983/solr/admin/metrics?group=core&prefix=CACHE.searcher.documentCache`.
-
=== Query Result Cache
This cache holds the results of previous searches: ordered lists of document
IDs based on a query, a sort, and the range of documents requested
-*Registry and Path:* `solr.<core>:CACHE.searcher.queryResultCache`
-
-You can get the statistics shown in the table below with an API request such
as
`\http://localhost:8983/solr/admin/metrics?group=core&prefix=CACHE.searcher.queryResultCache`.
+You can get the statistics shown in the table below with an API request such
as `\http://localhost:8983/solr/admin/metrics?category=CACHE`
Review Comment:
there is no table below. And don't bother, please :-)
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc:
##########
@@ -178,6 +178,8 @@ Nowadays, the HTTP request is available via internal APIs:
`SolrQueryRequest.get
+
+NOTE: The previous parse-context-based configuration (`parseContext.config`)
is no longer supported. Tika parser-specific properties must now be configured
directly on the Tika Server itself, rather than through Solr configuration.
Please refer to the Tika Server documentation for details on how to set these
properties.
+* SolrInfoMBeanHandler and PluginInfoHandler have been removed
Review Comment:
People will be more aware of the path endpoint than internal Java classes.
It's a bit odd to not see other things listed in the removal section but are
referenced below under OTEL. Please add a general reference below to the OTEL
section or list them or move them.
The "<metrics>" configuration in solr.xml should have been removed, even
though you forgot to literally remove it. It does nothing right now. It's no
longer possible to disable metrics.
##########
solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc:
##########
@@ -186,11 +188,21 @@ Nowadays, the HTTP request is available via internal
APIs: `SolrQueryRequest.get
Solr upgraded to Jetty 12.x from 10.x as Jetty 10 and 11 have reached
end-of-life support. Jetty 12.x requires Java 17 or newer and is fully
compatible with Solr's new minimum requirement of Java 21. This upgrade brings
support for modern HTTP protocols and adopts the Jakarta EE 10 namespace. For
more details, see https://webtide.com/jetty-12-has-arrived/.
-=== Open Telemetry
+=== OpenTelemetry
+
+Solr 10 introduces a major overhaul by migrating from Dropwizard metrics to
OpenTelemetry (OTEL) for observability. This migration provides native
Prometheus support, OTLP support, exemplar support for tracing correlation, and
native attributes and labels on all metrics.
Review Comment:
"introduces a major overhaul" doesn't sound right. I'd just skip those
words.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
