[
https://issues.apache.org/jira/browse/HDDS-16136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16136:
------------------------------
Description:
SCM HA groups whose Ratis peers were recorded with IPv4 literal addresses
cannot move to IPv6-only operation by editing ozone-site.xml alone. SCM starts
its Ratis server with an empty group and recovers the peers, including their
addresses, from Raft storage, so the stored IPv4 addresses keep being used.
This is not a blocker for IPv6 support in general. It does not affect:
* New clusters deployed with IPv6 addresses.
* Peers configured with DNS names, which remain the preferred migration path
because the stored hostname resolves to IPv6.
* Dual-stack clusters that keep IPv4 connectivity.
* OM HA, which passes its configured peers to Ratis at startup, so its
connections follow ozone-site.xml. Only the stored and displayed addresses may
stay stale. Enabling the Ratis startup option below for OM
({{ozone.om.ha.raft.server.startup.refresh-peer-addresses.enabled}}) refreshes
them without code changes.
Recommended approach:
* Use a Ratis startup option that refreshes the stored peer addresses from the
peers supplied at startup (RATIS-2754), as proposed in the review of
[apache/ratis#1630|https://github.com/apache/ratis/pull/1630] (RATIS-2082 was
closed as Won't Do). SCM would pass its configured peers when starting its
Ratis server and enable the option, so ozone-site.xml becomes the single source
for both the addresses SCM connects to and the addresses Ratis stores. A wrong
address is fixed by correcting the configuration and restarting. This needs the
option in a Ratis release and an Ozone upgrade to that release.
The option matches startup peers by peer ID. SCM peer IDs are SCM UUIDs
({{scmUuid}} in each SCM's VERSION file), which ozone-site.xml does not
contain, so SCM also needs to map each configured SCM node ID to its peer ID:
* On each startup, match the configured Ratis addresses to the peers in the
recovered Raft configuration and persist the node ID to SCM UUID mapping in the
VERSION file. This works while the addresses still match, so the mapping is in
place before the addresses change.
* When the option is enabled and the group has a persisted Raft configuration,
start with the local peer and the mapped peers, using the addresses from
ozone-site.xml. A peer without a mapping keeps its stored address.
* Keep starting a newly bootstrapped SCM with an empty group, so it still joins
through addSCM.
Migration then becomes: upgrade with the existing addresses, change the
addresses in ozone-site.xml, enable the option and do a rolling restart.
Alternatives considered:
* Register the configured SCM peers with Ratis before the server starts,
without the startup option. This is an Ozone only change, but it relies on
Ratis keeping the first address registered for a peer, and the stored and
displayed addresses would stay stale and could be propagated by later SCM
membership changes.
* Replace each SCM through decommission and bootstrap with its new address.
This works with existing tools but takes many steps per SCM and reduces fault
tolerance while it runs.
* Run the offline ratis-shell {{local raftMetaConf}} command ({{ozone admin
ratis local raftMetaConf}}, HDDS-10479) on each stopped SCM, passing each peer
as {{<SCM UUID>|<new address>}} with the UUIDs from {{ozone admin scm roles}}.
This works today but is manual and error prone, and a wrong address leaves that
SCM unable to rejoin. It can be documented as an interim procedure until the
startup option is available.
Requirements:
* Preserve the Ratis group ID, peer IDs, storage, and committed state.
* Support a rolling restart while both IPv4 and IPv6 addresses are reachable.
* Allow a wrong address to be corrected by fixing the configuration and
restarting.
* Validate leader election and failover after migration.
* Add SCM migration tests.
Parent epic: HDDS-15763.
was:
SCM HA groups whose Ratis peers were recorded with IPv4 literal addresses
cannot move to IPv6-only operation by editing ozone-site.xml alone. SCM starts
its Ratis server with an empty group and recovers the peers, including their
addresses, from Raft storage, so the stored IPv4 addresses keep being used.
This is not a blocker for IPv6 support in general. It does not affect:
* New clusters deployed with IPv6 addresses.
* Peers configured with DNS names, which remain the preferred migration path
because the stored hostname resolves to IPv6.
* Dual-stack clusters that keep IPv4 connectivity.
* OM HA, which passes its configured peers to Ratis at startup, so its
connections follow ozone-site.xml. Only the stored and displayed addresses may
stay stale.
Recommended approach:
* Use a Ratis startup option that refreshes the stored peer addresses from the
peers supplied at startup, as proposed in the review of
[apache/ratis#1630|https://github.com/apache/ratis/pull/1630] (RATIS-2082 was
closed as Won't Do). SCM would pass its configured peers when starting its
Ratis server and enable the option, so ozone-site.xml becomes the single source
for both the addresses SCM connects to and the addresses Ratis stores. A wrong
address is fixed by correcting the configuration and restarting. This needs the
option in a Ratis release and an Ozone upgrade to that release.
Alternatives considered:
* Register the configured SCM peers with Ratis before the server starts,
without the startup option. This is an Ozone only change, but it relies on
Ratis keeping the first address registered for a peer, and the stored and
displayed addresses would stay stale and could be propagated by later SCM
membership changes.
* Replace each SCM through decommission and bootstrap with its new address.
This works with existing tools but takes many steps per SCM and reduces fault
tolerance while it runs.
* Run the offline ratis-shell {{local raftMetaConf}} command on each stopped
SCM. This works today but is manual and error prone, and a wrong address leaves
that SCM unable to rejoin. It can be documented as an interim procedure until
the startup option is available.
Requirements:
* Preserve the Ratis group ID, peer IDs, storage, and committed state.
* Support a rolling restart while both IPv4 and IPv6 addresses are reachable.
* Allow a wrong address to be corrected by fixing the configuration and
restarting.
* Validate leader election and failover after migration.
* Add SCM migration tests.
Parent epic: HDDS-15763.
> Support migrating SCM HA from IPv4 literal Ratis peer addresses to IPv6-only
> ----------------------------------------------------------------------------
>
> Key: HDDS-16136
> URL: https://issues.apache.org/jira/browse/HDDS-16136
> Project: Apache Ozone
> Issue Type: Task
> Reporter: Siyao Meng
> Priority: Major
>
> SCM HA groups whose Ratis peers were recorded with IPv4 literal addresses
> cannot move to IPv6-only operation by editing ozone-site.xml alone. SCM
> starts its Ratis server with an empty group and recovers the peers, including
> their addresses, from Raft storage, so the stored IPv4 addresses keep being
> used.
> This is not a blocker for IPv6 support in general. It does not affect:
> * New clusters deployed with IPv6 addresses.
> * Peers configured with DNS names, which remain the preferred migration path
> because the stored hostname resolves to IPv6.
> * Dual-stack clusters that keep IPv4 connectivity.
> * OM HA, which passes its configured peers to Ratis at startup, so its
> connections follow ozone-site.xml. Only the stored and displayed addresses
> may stay stale. Enabling the Ratis startup option below for OM
> ({{ozone.om.ha.raft.server.startup.refresh-peer-addresses.enabled}})
> refreshes them without code changes.
> Recommended approach:
> * Use a Ratis startup option that refreshes the stored peer addresses from
> the peers supplied at startup (RATIS-2754), as proposed in the review of
> [apache/ratis#1630|https://github.com/apache/ratis/pull/1630] (RATIS-2082 was
> closed as Won't Do). SCM would pass its configured peers when starting its
> Ratis server and enable the option, so ozone-site.xml becomes the single
> source for both the addresses SCM connects to and the addresses Ratis stores.
> A wrong address is fixed by correcting the configuration and restarting. This
> needs the option in a Ratis release and an Ozone upgrade to that release.
> The option matches startup peers by peer ID. SCM peer IDs are SCM UUIDs
> ({{scmUuid}} in each SCM's VERSION file), which ozone-site.xml does not
> contain, so SCM also needs to map each configured SCM node ID to its peer ID:
> * On each startup, match the configured Ratis addresses to the peers in the
> recovered Raft configuration and persist the node ID to SCM UUID mapping in
> the VERSION file. This works while the addresses still match, so the mapping
> is in place before the addresses change.
> * When the option is enabled and the group has a persisted Raft
> configuration, start with the local peer and the mapped peers, using the
> addresses from ozone-site.xml. A peer without a mapping keeps its stored
> address.
> * Keep starting a newly bootstrapped SCM with an empty group, so it still
> joins through addSCM.
> Migration then becomes: upgrade with the existing addresses, change the
> addresses in ozone-site.xml, enable the option and do a rolling restart.
> Alternatives considered:
> * Register the configured SCM peers with Ratis before the server starts,
> without the startup option. This is an Ozone only change, but it relies on
> Ratis keeping the first address registered for a peer, and the stored and
> displayed addresses would stay stale and could be propagated by later SCM
> membership changes.
> * Replace each SCM through decommission and bootstrap with its new address.
> This works with existing tools but takes many steps per SCM and reduces fault
> tolerance while it runs.
> * Run the offline ratis-shell {{local raftMetaConf}} command ({{ozone admin
> ratis local raftMetaConf}}, HDDS-10479) on each stopped SCM, passing each
> peer as {{<SCM UUID>|<new address>}} with the UUIDs from {{ozone admin scm
> roles}}. This works today but is manual and error prone, and a wrong address
> leaves that SCM unable to rejoin. It can be documented as an interim
> procedure until the startup option is available.
> Requirements:
> * Preserve the Ratis group ID, peer IDs, storage, and committed state.
> * Support a rolling restart while both IPv4 and IPv6 addresses are reachable.
> * Allow a wrong address to be corrected by fixing the configuration and
> restarting.
> * Validate leader election and failover after migration.
> * Add SCM migration tests.
> Parent epic: HDDS-15763.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
