[
https://issues.apache.org/jira/browse/HDDS-13112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17953553#comment-17953553
]
Wei-Chiu Chuang commented on HDDS-13112:
----------------------------------------
ChatGPT suggests the following update:
{noformat}
## Automatic Snapshot Installation for Stale Ozone Managers
In an Ozone Manager (OM) High Availability (HA) cluster, all OM nodes maintain
a consistent metadata state using the Ratis consensus protocol. Sometimes, an
OM follower node may be offline or fall so far behind the leader OM’s log that
it cannot catch up by replaying individual log entries.
Starting from Ozone version X.Y.Z (update with the actual version introduced),
the OM HA implementation includes an **automatic snapshot installation and
recovery process** for such cases:
- **Snapshot Installation Trigger:**
When a follower OM falls significantly behind and is unable to catch up with
the leader OM through standard log replication, the leader OM will notify the
follower to install a snapshot. This is handled internally by the OM state
machine.
- **How it works:**
- The follower OM receives a snapshot installation notification from the
leader via the consensus protocol.
- The follower OM then downloads and installs the latest consistent
checkpoint (snapshot) from the leader OM.
- After installing the snapshot, the follower OM resumes normal operation and
log replication from the new state.
- **Relevant Implementation:**
This logic is implemented in the
`OzoneManagerStateMachine.notifyInstallSnapshotFromLeader()` method. The
install is triggered automatically by the consensus layer (Ratis) when it
detects that a follower cannot catch up by log replay alone.
- **What this means for administrators:**
- In most scenarios, stale OMs will recover automatically after coming back
online, even if they have missed a large number of operations.
- Manual intervention (such as running `ozone om --bootstrap`) is only
required when adding a new OM node to the cluster or when explicitly requested
by support instructions.
**References:**
- [OzoneManagerStateMachine.notifyInstallSnapshotFromLeader source
code](https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java)
- [Ozone Ratis HA design](https://ozone.apache.org/docs/edge/design/om-ha.html)
> **Note:** This mechanism provides better resilience and automated recovery
> for OM nodes that have been offline for extended periods.
{noformat}
> [Docs] OM Bootstrap can also happen when follower falls behind too much
> -----------------------------------------------------------------------
>
> Key: HDDS-13112
> URL: https://issues.apache.org/jira/browse/HDDS-13112
> Project: Apache Ozone
> Issue Type: Improvement
> Components: documentation, OM HA
> Affects Versions: 2.1.0
> Reporter: Wei-Chiu Chuang
> Priority: Minor
>
> [https://ozone.apache.org/docs/edge/feature/om-ha.html]
>
> It is written in the user doc that OM bootstrap happens when adding a new OM.
> Ratis would trigger notifyInstallSnapshotFromLeader() if a follower OM falls
> behind the leader OM. We should update the doc to include this condition too.
> [https://github.com/apache/ratis/blob/c1da37cb455bbf94da267b3f2b9bf3884017e1ca/ratis-server-api/src/main/java/org/apache/ratis/server/leader/LogAppender.java#L109]
> {code:java}
> /**
> * Should this {@link LogAppender} send a snapshot to the follower?
> *
> * @return the snapshot if it should install a snapshot; otherwise, return
> null.
> */
> default SnapshotInfo shouldInstallSnapshot() {
> // we should install snapshot if the follower needs to catch up and:
> // 1. there is no local log entry but there is snapshot
> // 2. or the follower's next index is smaller than the log start index
> // 3. or the follower is bootstrapping and has not installed any snapshot
> yet {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
