gerlowskija commented on code in PR #4886: URL: https://github.com/apache/solr/pull/4886#discussion_r3957725489
########## solr/solr-ref-guide/modules/deployment-guide/pages/healthcheck.adoc: ########## @@ -0,0 +1,181 @@ += Healthchecks +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +Solr provides two distinct tools for checking whether things are working, and they answer different questions. +Confusing them is easy, since both are casually called a "healthcheck", so this page describes each and helps you pick the right one. + +[cols="1,2,2"] +|=== +| |Node Health Endpoint |`bin/solr healthcheck` command + +|Answers +|"Is this node alive and part of the cluster?" +|"Does this collection actually have data, and is every replica in sync?" + +|Scope +|A single node +|A single collection, across all its shards and replicas + +|Checks the index/data? +|No -- metadata only +|Yes -- runs real queries + +|Interface +|HTTP endpoint +|Command-line tool + +|Typical use +|Load balancer / orchestrator liveness or readiness probe +|Manual or scripted diagnosis after deployment, restart, or an incident +|=== + +== Node Health Endpoint + +The `api/node/health` endpoint ({solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth]) reports whether a single Solr node is alive and able to participate in the cluster. +It is designed to be cheap enough to poll frequently from a load balancer or an orchestrator like Kubernetes. + +[IMPORTANT] +==== +None of these checks touch the index. +A node with a collection that has zero documents, or a collection whose query results are wrong, will still report a healthy `200 OK` here. +If you need to confirm a collection actually has data, see <<Collection Data Healthcheck>> below. +==== + +What the endpoint checks -- and how it fails -- depends on whether the node is running in SolrCloud mode or user-managed (leader-follower) mode. + +=== SolrCloud Mode + +In SolrCloud mode, the endpoint returns HTTP `200 OK` only if all of the following are true, and HTTP `503 Unavailable` otherwise: + +* The node's core container has finished starting up. +* The node is connected to ZooKeeper. +* The node is listed in ZooKeeper's `live_nodes`. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health"; +---- + +Healthy node: + +[source,json] +---- +{ + "responseHeader":{"status":0,"QTime":1}, + "status":"OK" +} +---- + +Unhealthy node (HTTP 503): + +[source,json] +---- +{ + "responseHeader":{"status":503,"QTime":1}, + "status":"FAILURE", + "error":{"msg":"Not connected to ZK"} +} +---- + +==== Rolling Restarts: `requireHealthyCores` + +Add `requireHealthyCores=true` to additionally require that every local replica belonging to an active shard has finished initializing -- i.e., none are in the `RECOVERING` or `DOWN` state. +This is useful as a readiness probe during a rolling restart, so an orchestrator doesn't move on to the next node while the one it just restarted still has replicas recovering. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health?requireHealthyCores=true"; +---- + +=== User-Managed (Leader-Follower) Mode + +In user-managed replication, the endpoint instead checks how far behind each local follower core is from its leader, in Lucene commit generations. +Set `maxGenerationLag=<n>` to fail the health check once a follower falls more than `<n>` generations behind; without it, the check simply reports `OK` once a follower has replicated at least once, even if it later falls arbitrarily far behind. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health?maxGenerationLag=100"; +---- + +Healthy node (all followers within the allowed lag): + +[source,json] +---- +{ + "responseHeader":{"status":0,"QTime":1}, + "status":"OK" +} +---- + +Unhealthy node (a follower has fallen too far behind, HTTP 503): + +[source,json] +---- +{ + "responseHeader":{"status":503,"QTime":1}, + "status":"FAILURE", + "error":{"msg":"Cores violating maxGenerationLag:100.\nCore collection1 is lagging by 137 generations"} +} +---- + +See xref:user-managed-index-replication.adoc#monitoring-follower-replication-lag[Monitoring Follower Replication Lag] for details. + +== Collection Data Healthcheck Review Comment: [0] I found this section-title a bit confusing, as it didn't appear in the summary table at the top of the page. Might be worth aligning that terminology. ########## solr/solr-ref-guide/modules/configuration-guide/pages/implicit-requesthandlers.adoc: ########## @@ -40,24 +40,8 @@ This handler must have a collection name in the path to the endpoint. |`solr/<collection>/admin/file` |{solr-javadocs}/core/org/apache/solr/handler/admin/ShowFileRequestHandler.html[ShowFileRequestHandler] |`_ADMIN_FILE` |=== -Health:: Report the health of the node. -+ -[cols="3*.",frame=none,grid=cols,options="header"] -|=== -|API Endpoints |Class & Javadocs |Paramset -|v1: `solr/admin/info/health` - -v2: `api/node/health` |v1: {solr-javadocs}/core/org/apache/solr/handler/admin/HealthCheckHandler.html[HealthCheckHandler] - -v2: {solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth] | -|=== -+ -In SolrCloud mode the handler checks that the node is connected to ZooKeeper and is listed in live nodes. -The optional `requireHealthyCores=true` parameter additionally requires that all local replicas be in an active state, which is useful for rolling-restart probes. -+ -In user-managed (leader-follower) mode the handler checks replication lag. -The optional `maxGenerationLag=<n>` parameter specifies the maximum number of Lucene commit generations by which a follower is allowed to trail its leader; the endpoint returns HTTP 503 if any core exceeds this threshold. -See xref:deployment-guide:user-managed-index-replication.adoc#monitoring-follower-replication-lag[Monitoring Follower Replication Lag] for details and examples. +Health:: Report the health of the node, via the `api/node/health` endpoint ({solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth]). +See xref:deployment-guide:healthcheck.adoc[] for details, parameters, and examples. Review Comment: [Q] Is there a reason you dropped the references to the v1 endpoint here? ########## solr/solr-ref-guide/modules/deployment-guide/pages/healthcheck.adoc: ########## @@ -0,0 +1,181 @@ += Healthchecks +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +Solr provides two distinct tools for checking whether things are working, and they answer different questions. +Confusing them is easy, since both are casually called a "healthcheck", so this page describes each and helps you pick the right one. + +[cols="1,2,2"] +|=== +| |Node Health Endpoint |`bin/solr healthcheck` command + +|Answers +|"Is this node alive and part of the cluster?" +|"Does this collection actually have data, and is every replica in sync?" + +|Scope +|A single node +|A single collection, across all its shards and replicas + +|Checks the index/data? +|No -- metadata only +|Yes -- runs real queries + +|Interface +|HTTP endpoint +|Command-line tool + +|Typical use +|Load balancer / orchestrator liveness or readiness probe +|Manual or scripted diagnosis after deployment, restart, or an incident +|=== + +== Node Health Endpoint + +The `api/node/health` endpoint ({solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth]) reports whether a single Solr node is alive and able to participate in the cluster. +It is designed to be cheap enough to poll frequently from a load balancer or an orchestrator like Kubernetes. + +[IMPORTANT] +==== +None of these checks touch the index. +A node with a collection that has zero documents, or a collection whose query results are wrong, will still report a healthy `200 OK` here. +If you need to confirm a collection actually has data, see <<Collection Data Healthcheck>> below. +==== + +What the endpoint checks -- and how it fails -- depends on whether the node is running in SolrCloud mode or user-managed (leader-follower) mode. + +=== SolrCloud Mode + +In SolrCloud mode, the endpoint returns HTTP `200 OK` only if all of the following are true, and HTTP `503 Unavailable` otherwise: + +* The node's core container has finished starting up. +* The node is connected to ZooKeeper. +* The node is listed in ZooKeeper's `live_nodes`. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health"; +---- + +Healthy node: + +[source,json] +---- +{ + "responseHeader":{"status":0,"QTime":1}, + "status":"OK" +} +---- + +Unhealthy node (HTTP 503): + +[source,json] +---- +{ + "responseHeader":{"status":503,"QTime":1}, + "status":"FAILURE", + "error":{"msg":"Not connected to ZK"} +} +---- + +==== Rolling Restarts: `requireHealthyCores` + +Add `requireHealthyCores=true` to additionally require that every local replica belonging to an active shard has finished initializing -- i.e., none are in the `RECOVERING` or `DOWN` state. +This is useful as a readiness probe during a rolling restart, so an orchestrator doesn't move on to the next node while the one it just restarted still has replicas recovering. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health?requireHealthyCores=true"; +---- + +=== User-Managed (Leader-Follower) Mode + +In user-managed replication, the endpoint instead checks how far behind each local follower core is from its leader, in Lucene commit generations. +Set `maxGenerationLag=<n>` to fail the health check once a follower falls more than `<n>` generations behind; without it, the check simply reports `OK` once a follower has replicated at least once, even if it later falls arbitrarily far behind. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health?maxGenerationLag=100"; +---- + +Healthy node (all followers within the allowed lag): + +[source,json] +---- +{ + "responseHeader":{"status":0,"QTime":1}, + "status":"OK" +} +---- + +Unhealthy node (a follower has fallen too far behind, HTTP 503): + +[source,json] +---- +{ + "responseHeader":{"status":503,"QTime":1}, + "status":"FAILURE", + "error":{"msg":"Cores violating maxGenerationLag:100.\nCore collection1 is lagging by 137 generations"} +} +---- + +See xref:user-managed-index-replication.adoc#monitoring-follower-replication-lag[Monitoring Follower Replication Lag] for details. + +== Collection Data Healthcheck + +Where the Node Health endpoint asks "is this process alive?", `bin/solr healthcheck` asks "does this collection actually work?" +It queries the collection for a total document count, then queries each individual replica directly (non-distributed) for its own document count, and reports whether every shard has a leader and every replica is `ACTIVE`. + +Because it issues real queries against real replicas, it's a heavier operation than the Node Health endpoint, and it's a CLI tool rather than an HTTP endpoint -- it isn't meant to be polled continuously by a load balancer. +Reach for it after a deploy, a restart, or an incident, when you want to confirm a specific collection is actually intact, not just that the nodes are up. + +[source,bash] +---- +bin/solr healthcheck -c gettingstarted +---- + +[source,json] +---- +{ + "collection":"gettingstarted", + "status":"healthy", + "numDocs":42, + "numShards":2, + "shards":[ + { + "shard":"shard1", + "status":"healthy", + "replicas":[ + {"name":"core_node1", "url":"...", "numDocs":21, "status":"active", "leader":true} + ] + } + ] +} +---- + +See xref:solr-control-script-reference.adoc#healthcheck[the `healthcheck` command reference] for the full list of options (ZooKeeper vs. Solr URL connection, credentials, etc.). + +== Coming From `/admin/ping`? + +Solr's older `/admin/ping` endpoint (`PingRequestHandler`) was deprecated in Solr 10.1 and removed in Solr 11. +Unlike the Node Health endpoint, Ping executed a real, configurable query against a specific core -- so a passing ping meant the query path was actually functional, not just that the node was alive. + +If you configured a load balancer against `/admin/ping` specifically to catch a broken query path (not just a dead node), the Node Health endpoint is *not* a drop-in replacement -- it never touches the index. Review Comment: [0] I personally don't know what a "broken query path" means. Maybe I'm just blanking, but if you end up retaining this section (see my comment on L174 above) then might be worth word-smithing that a bit. ########## solr/solr-ref-guide/modules/deployment-guide/pages/healthcheck.adoc: ########## @@ -0,0 +1,181 @@ += Healthchecks +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +Solr provides two distinct tools for checking whether things are working, and they answer different questions. +Confusing them is easy, since both are casually called a "healthcheck", so this page describes each and helps you pick the right one. + +[cols="1,2,2"] +|=== +| |Node Health Endpoint |`bin/solr healthcheck` command + +|Answers +|"Is this node alive and part of the cluster?" +|"Does this collection actually have data, and is every replica in sync?" + +|Scope +|A single node +|A single collection, across all its shards and replicas + +|Checks the index/data? +|No -- metadata only +|Yes -- runs real queries + +|Interface +|HTTP endpoint +|Command-line tool + +|Typical use +|Load balancer / orchestrator liveness or readiness probe +|Manual or scripted diagnosis after deployment, restart, or an incident +|=== + +== Node Health Endpoint + +The `api/node/health` endpoint ({solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth]) reports whether a single Solr node is alive and able to participate in the cluster. +It is designed to be cheap enough to poll frequently from a load balancer or an orchestrator like Kubernetes. + +[IMPORTANT] +==== +None of these checks touch the index. +A node with a collection that has zero documents, or a collection whose query results are wrong, will still report a healthy `200 OK` here. +If you need to confirm a collection actually has data, see <<Collection Data Healthcheck>> below. +==== + +What the endpoint checks -- and how it fails -- depends on whether the node is running in SolrCloud mode or user-managed (leader-follower) mode. + +=== SolrCloud Mode + +In SolrCloud mode, the endpoint returns HTTP `200 OK` only if all of the following are true, and HTTP `503 Unavailable` otherwise: + +* The node's core container has finished starting up. +* The node is connected to ZooKeeper. +* The node is listed in ZooKeeper's `live_nodes`. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health"; +---- + +Healthy node: + +[source,json] +---- +{ + "responseHeader":{"status":0,"QTime":1}, + "status":"OK" +} +---- + +Unhealthy node (HTTP 503): + +[source,json] +---- +{ + "responseHeader":{"status":503,"QTime":1}, + "status":"FAILURE", + "error":{"msg":"Not connected to ZK"} +} +---- + +==== Rolling Restarts: `requireHealthyCores` + +Add `requireHealthyCores=true` to additionally require that every local replica belonging to an active shard has finished initializing -- i.e., none are in the `RECOVERING` or `DOWN` state. +This is useful as a readiness probe during a rolling restart, so an orchestrator doesn't move on to the next node while the one it just restarted still has replicas recovering. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health?requireHealthyCores=true"; +---- + +=== User-Managed (Leader-Follower) Mode + +In user-managed replication, the endpoint instead checks how far behind each local follower core is from its leader, in Lucene commit generations. +Set `maxGenerationLag=<n>` to fail the health check once a follower falls more than `<n>` generations behind; without it, the check simply reports `OK` once a follower has replicated at least once, even if it later falls arbitrarily far behind. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health?maxGenerationLag=100"; +---- + +Healthy node (all followers within the allowed lag): + +[source,json] +---- +{ + "responseHeader":{"status":0,"QTime":1}, + "status":"OK" +} +---- + +Unhealthy node (a follower has fallen too far behind, HTTP 503): + +[source,json] +---- +{ + "responseHeader":{"status":503,"QTime":1}, + "status":"FAILURE", + "error":{"msg":"Cores violating maxGenerationLag:100.\nCore collection1 is lagging by 137 generations"} +} +---- + +See xref:user-managed-index-replication.adoc#monitoring-follower-replication-lag[Monitoring Follower Replication Lag] for details. + +== Collection Data Healthcheck + +Where the Node Health endpoint asks "is this process alive?", `bin/solr healthcheck` asks "does this collection actually work?" +It queries the collection for a total document count, then queries each individual replica directly (non-distributed) for its own document count, and reports whether every shard has a leader and every replica is `ACTIVE`. + +Because it issues real queries against real replicas, it's a heavier operation than the Node Health endpoint, and it's a CLI tool rather than an HTTP endpoint -- it isn't meant to be polled continuously by a load balancer. +Reach for it after a deploy, a restart, or an incident, when you want to confirm a specific collection is actually intact, not just that the nodes are up. + +[source,bash] +---- +bin/solr healthcheck -c gettingstarted +---- + +[source,json] +---- +{ + "collection":"gettingstarted", + "status":"healthy", + "numDocs":42, + "numShards":2, + "shards":[ + { + "shard":"shard1", + "status":"healthy", + "replicas":[ + {"name":"core_node1", "url":"...", "numDocs":21, "status":"active", "leader":true} + ] + } + ] +} +---- + +See xref:solr-control-script-reference.adoc#healthcheck[the `healthcheck` command reference] for the full list of options (ZooKeeper vs. Solr URL connection, credentials, etc.). + +== Coming From `/admin/ping`? + +Solr's older `/admin/ping` endpoint (`PingRequestHandler`) was deprecated in Solr 10.1 and removed in Solr 11. Review Comment: [-1] IMO it does not make sense to continue to carry around documentation for features that have been removed from Solr. We don't do it for DIH, etc. These sort of "If you used to use 'X', switch to 'Y'" notices are one of the main benefits of our changelog. IMO this should just go there. ########## solr/solr-ref-guide/modules/deployment-guide/pages/healthcheck.adoc: ########## @@ -0,0 +1,181 @@ += Healthchecks +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +Solr provides two distinct tools for checking whether things are working, and they answer different questions. +Confusing them is easy, since both are casually called a "healthcheck", so this page describes each and helps you pick the right one. + +[cols="1,2,2"] +|=== +| |Node Health Endpoint |`bin/solr healthcheck` command + +|Answers +|"Is this node alive and part of the cluster?" +|"Does this collection actually have data, and is every replica in sync?" + +|Scope +|A single node +|A single collection, across all its shards and replicas + +|Checks the index/data? +|No -- metadata only +|Yes -- runs real queries + +|Interface +|HTTP endpoint +|Command-line tool + +|Typical use +|Load balancer / orchestrator liveness or readiness probe +|Manual or scripted diagnosis after deployment, restart, or an incident +|=== + +== Node Health Endpoint + +The `api/node/health` endpoint ({solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth]) reports whether a single Solr node is alive and able to participate in the cluster. +It is designed to be cheap enough to poll frequently from a load balancer or an orchestrator like Kubernetes. + +[IMPORTANT] +==== +None of these checks touch the index. +A node with a collection that has zero documents, or a collection whose query results are wrong, will still report a healthy `200 OK` here. +If you need to confirm a collection actually has data, see <<Collection Data Healthcheck>> below. +==== + +What the endpoint checks -- and how it fails -- depends on whether the node is running in SolrCloud mode or user-managed (leader-follower) mode. + +=== SolrCloud Mode + +In SolrCloud mode, the endpoint returns HTTP `200 OK` only if all of the following are true, and HTTP `503 Unavailable` otherwise: + +* The node's core container has finished starting up. +* The node is connected to ZooKeeper. +* The node is listed in ZooKeeper's `live_nodes`. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health"; +---- + +Healthy node: + +[source,json] +---- +{ + "responseHeader":{"status":0,"QTime":1}, + "status":"OK" +} +---- + +Unhealthy node (HTTP 503): + +[source,json] +---- +{ + "responseHeader":{"status":503,"QTime":1}, + "status":"FAILURE", + "error":{"msg":"Not connected to ZK"} +} +---- + +==== Rolling Restarts: `requireHealthyCores` + +Add `requireHealthyCores=true` to additionally require that every local replica belonging to an active shard has finished initializing -- i.e., none are in the `RECOVERING` or `DOWN` state. +This is useful as a readiness probe during a rolling restart, so an orchestrator doesn't move on to the next node while the one it just restarted still has replicas recovering. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health?requireHealthyCores=true"; +---- + +=== User-Managed (Leader-Follower) Mode + +In user-managed replication, the endpoint instead checks how far behind each local follower core is from its leader, in Lucene commit generations. +Set `maxGenerationLag=<n>` to fail the health check once a follower falls more than `<n>` generations behind; without it, the check simply reports `OK` once a follower has replicated at least once, even if it later falls arbitrarily far behind. + +[source,bash] +---- +curl "http://localhost:8983/api/node/health?maxGenerationLag=100"; +---- + +Healthy node (all followers within the allowed lag): + +[source,json] +---- +{ + "responseHeader":{"status":0,"QTime":1}, + "status":"OK" +} +---- + +Unhealthy node (a follower has fallen too far behind, HTTP 503): + +[source,json] +---- +{ + "responseHeader":{"status":503,"QTime":1}, + "status":"FAILURE", + "error":{"msg":"Cores violating maxGenerationLag:100.\nCore collection1 is lagging by 137 generations"} +} +---- + +See xref:user-managed-index-replication.adoc#monitoring-follower-replication-lag[Monitoring Follower Replication Lag] for details. + +== Collection Data Healthcheck + +Where the Node Health endpoint asks "is this process alive?", `bin/solr healthcheck` asks "does this collection actually work?" +It queries the collection for a total document count, then queries each individual replica directly (non-distributed) for its own document count, and reports whether every shard has a leader and every replica is `ACTIVE`. Review Comment: [Q] Is `bin/solr healthcheck` specific to SolrCloud? What would it do in stand-alone mode? [Q] What is a user supposed to do with this per-replica docCount info? Healthy, well-functioning replicas often disagree on doc count just by nature of their autoCommit periods not being perfectly aligned. It might be worth flagging that for users, so they don't treat mismatches as "issues". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
