[
https://issues.apache.org/jira/browse/HDDS-17122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17122:
------------------------------
Description:
OM HA and SCM HA have no hard limit on the number of nodes:
{{ozone.om.nodes.<serviceId>}} / {{ozone.scm.nodes.<serviceId>}} accept any
number of node IDs, and every node becomes a Ratis peer. In practice only
3-node rings are tested and documented (SCM-HA.md: "For reliable HA support
choose 3 independent nodes"). Operators ask for 5 nodes so a ring keeps HA
while one node is down for maintenance (see HDDS-3099).
This umbrella tracks making 5+ node OM and SCM rings a tested, documented
configuration.
h3. Findings so far
New MiniOzoneCluster integration tests with 5 OMs and 5 SCMs (HDDS-17123) show
that the core Ratis behaviour works: a single Raft group of 5, replication to
all nodes, leader transfer, surviving 2 failures including the leader, no
leader with 3 of 5 down, catch-up through checkpoint install, bootstrap 3 -> 5
and decommission 5 -> 3, secure 5-SCM failover.
The problems are around it:
* *Blocker:* HDDS-16924. An SCM that has installed a checkpoint can no longer
serve checkpoints once it is leader, so lagging followers never catch up. With
3 SCMs this is rare (the primordial SCM usually stays leader); with 5 SCMs and
a few failovers the 5-node test hits it reliably.
* Leadership transfer takes 4-13 s and the OM admin RPC is not idempotent under
client retry.
* OM clients cannot reach a leader that is not in their configured OM list.
* SCM checkpoint download is retried without backoff.
* Test infrastructure bugs in MiniOzoneHAClusterImpl.
* Missing acceptance coverage and documentation.
Each item is a sub-task.
was:
OM HA and SCM HA have no hard limit on the number of nodes:
{{ozone.om.nodes.<serviceId>}} / {{ozone.scm.nodes.<serviceId>}} accept any
number of node IDs, and every node becomes a Ratis peer. In practice only
3-node rings are tested and documented (SCM-HA.md: "For reliable HA support
choose 3 independent nodes"). Operators ask for 5 nodes so a ring keeps HA
while one node is down for maintenance (see HDDS-3099).
This umbrella tracks making 5+ node OM and SCM rings a tested, documented
configuration.
h3. Findings so far
New MiniOzoneCluster integration tests with 5 OMs and 5 SCMs (sub-task "Add
integration tests for 5-node OM HA and SCM HA") show that the core Ratis
behaviour works: a single Raft group of 5, replication to all nodes, leader
transfer, surviving 2 failures including the leader, no leader with 3 of 5
down, catch-up through checkpoint install, bootstrap 3 -> 5 and decommission 5
-> 3, secure 5-SCM failover.
The problems are around it:
* *Blocker:* HDDS-16924. An SCM that has installed a checkpoint can no longer
serve checkpoints once it is leader, so lagging followers never catch up. With
3 SCMs this is rare (the primordial SCM usually stays leader); with 5 SCMs and
a few failovers the 5-node test hits it reliably.
* Leadership transfer takes 4-13 s and the OM admin RPC is not idempotent under
client retry.
* OM clients cannot reach a leader that is not in their configured OM list.
* SCM checkpoint download is retried without backoff.
* Test infrastructure bugs in MiniOzoneHAClusterImpl.
* Missing acceptance coverage and documentation.
Each item is a sub-task.
> Support and test OM and SCM HA with more than 3 nodes
> -----------------------------------------------------
>
> Key: HDDS-17122
> URL: https://issues.apache.org/jira/browse/HDDS-17122
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM HA, SCM HA
> Reporter: Siyao Meng
> Priority: Major
>
> OM HA and SCM HA have no hard limit on the number of nodes:
> {{ozone.om.nodes.<serviceId>}} / {{ozone.scm.nodes.<serviceId>}} accept any
> number of node IDs, and every node becomes a Ratis peer. In practice only
> 3-node rings are tested and documented (SCM-HA.md: "For reliable HA support
> choose 3 independent nodes"). Operators ask for 5 nodes so a ring keeps HA
> while one node is down for maintenance (see HDDS-3099).
> This umbrella tracks making 5+ node OM and SCM rings a tested, documented
> configuration.
> h3. Findings so far
> New MiniOzoneCluster integration tests with 5 OMs and 5 SCMs (HDDS-17123)
> show that the core Ratis behaviour works: a single Raft group of 5,
> replication to all nodes, leader transfer, surviving 2 failures including the
> leader, no leader with 3 of 5 down, catch-up through checkpoint install,
> bootstrap 3 -> 5 and decommission 5 -> 3, secure 5-SCM failover.
> The problems are around it:
> * *Blocker:* HDDS-16924. An SCM that has installed a checkpoint can no longer
> serve checkpoints once it is leader, so lagging followers never catch up.
> With 3 SCMs this is rare (the primordial SCM usually stays leader); with 5
> SCMs and a few failovers the 5-node test hits it reliably.
> * Leadership transfer takes 4-13 s and the OM admin RPC is not idempotent
> under client retry.
> * OM clients cannot reach a leader that is not in their configured OM list.
> * SCM checkpoint download is retried without backoff.
> * Test infrastructure bugs in MiniOzoneHAClusterImpl.
> * Missing acceptance coverage and documentation.
> Each item is a sub-task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
