errose28 commented on code in PR #560: URL: https://github.com/apache/ozone-site/pull/560#discussion_r4138061284
########## docs/08-developer-guide/01-develop/05-upgrade-compatibility.md: ########## @@ -0,0 +1,280 @@ +--- +sidebar_label: Upgrade Compatibility +--- + +# Developing for Compatibility with Zero Downtime Upgrade + +Zero Downtime Upgrade (ZDU) lets the internal components of an Ozone cluster (OM, SCM, Datanode, S3 Gateway, and Recon) run in mixed versions during an upgrade with no interruption to service. To make this safe, the framework enforces one invariant that every developer commit must uphold: + +> While a rolling upgrade is in progress, components of the same type may be running different software but must behave as if they are the same version. Two OMs, two SCMs, or two Datanodes at different software versions must persist identical on-disk state and expose identical wire behavior until the cluster finalizes. + +Without this guarantee, a rolling upgrade can silently introduce behavior like diverging OM state machines, corrupting a container replica, or leaving a Datanode unable to re-register. This page is a practical guide to what counts as an incompatible change inside the cluster, and how to handle it in each area of the code. + +The rationale behind the framework lives in the [ZDU design document](https://github.com/apache/ozone/blob/master/hadoop-hdds/docs/content/design/zdu-design.md). Read it if you want the full model. This page is only focused on day-to-day development. + +:::note Scope +This page covers compatibility **between internal Ozone components** during an upgrade. External client ↔ Ozone-server compatibility is a separate concern. +::: + +## Background + +### Vocabulary + +- **Apparent version**: The version a component is *acting as*. It is persisted to disk and determines both the API the component exposes and the on-disk format it writes. + +- **Software version**: The highest component version contained in the running code. This is fixed by the bits that are installed. + +- **Pre-finalized**: The state a component is in when its apparent version on disk is *less than* its software version. All old features work; new features are blocked; downgrade is allowed. + +- **Finalized**: The state a component reaches when its apparent version *equals* its software version. All features are allowed; downgrade is no longer possible. + +In summary, any behavior incompatible with an old version's disk layout or API must be mapped to a version and disabled until its corresponding version is finalized. + +### Invariants + +To maintain consistency during zero downtime upgrade, the following invariants must be maintained: + +1. **Components of the same type always operate at the same apparent version** + - This means they expose the same API surface and persist data in the same format. +2. **For internal client/server relationships, the server is always upgraded before and finalizes before its client.** + - The only exception is Recon, which is a client of OM but is upgraded ahead of it. +3. **All internal components are running the newest software before finalization begins.** + +Only invariant 1 is the focus of this document which needs to be maintained by Ozone feature developers. Invariant 2 is maintained by the admin following the documented component upgrade order. Invariant 3 is loosely enforced by the upgrade framework built in to Ozone, but also falls on the admin to correctly execute. + +### Upgrade Order + +Components are upgraded in the following order to meet the client/server version invariant: + +- SCM +- Recon +- Datanodes +- OM +- S3 and HttpFS Gateways + +Feature developers do not need to account for upgrade orders that deviate from this when handling compatibility for a feature. An out of order upgrade is considered user error. + +### Finalization Order + +Components finalize in the same order they are upgraded. Finalization is triggered by a single admin command and the cluster ensures the correct finalization order internally. + +- SCM: Finalized via Ratis +- Datanodes: Finalized asynchronously by SCM after it finalizes itself. +- OM: Finalized via Ratis only once HDDS (SCM and all live Datanodes) have finalized + +Stateless gateway components do not need to finalize since they have no disk state to manage on downgrade and do not communicate with each other. Their compatibility is handled solely by the guarantee that the server they call remains backwards compatible with their client version. Recon currently finalizes immediately on upgrade. Downgrade support for Recon may be added in the future. + +## Quick Start + +Adding a feature means adding an entry to the affected components' version enums. That is where a contributor adding a change makes their first edit: + +- [`OzoneManagerVersion.java`](https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneManagerVersion.java) — used within the OM ring and reported to external clients. +- [`HDDSVersion.java`](https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HDDSVersion.java) — shared by SCM and Datanodes so SCM can orchestrate Datanode finalization and shared with clients. +- [`ReconVersion.java`](https://github.com/apache/ozone/blob/master/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/upgrade/ReconVersion.java) — Recon's own internal disk-format version which is currently not shared with other components. + +:::warning +Do **not** add new entries to `OMLayoutFeature` or `HDDSLayoutFeature`. Those enums are frozen after the migration to ZDU's new versioning framework. Every new version goes into `OzoneManagerVersion`, `HDDSVersion`, or `ReconVersion`. +::: + +```java +public enum OzoneManagerVersion implements ComponentVersion { + // Previous versions... + MY_NEW_FEATURE(101 /*or next available version integer*/, "My new feature"), +} +``` + +Most incompatible changes are handled by gating the new behavior on the component's apparent version through its version manager: + +```java +if (ozoneManager.getVersionManager().isAllowed(OzoneManagerVersion.MY_NEW_FEATURE)) { + // All capabilities of MY_NEW_FEATURE can be used, even if they are incompatible with older components. +} else { + // Must continue to act as the older version without using incompatible changes from MY_NEW_FEATURE. +} +``` + +`isAllowed(...)` returns true only once the component's apparent version is at least the version you name. Each component has a version manager with an equivalent method. + +:::warning +It is incorrect to write the following: + +```java +if (!ozoneManager.getVersionManager().needsFinalization()) { + // Use MY_NEW_FEATURE +} else { + // Bypass MY_NEW_FEATURE +} +``` + +New versions will continue be added in subsequent releases after `MY_NEW_FEATURE`. Say `MY_NEW_FEATURE_2` is added in the next release. When upgrading from software version `MY_NEW_FEATURE` to `MY_NEW_FEATURE_2`, `MY_NEW_FEATURE` must remain finalized and usable from the previous upgrade even if the component is not yet finalized for `MY_NEW_FEATURE_2`. +::: + +## Surface-by-surface guide + +Each surface below follows the same shape: **what is considered incompatible** and **the pattern to handle it**. Note that OM and SCM are finalized via Ratis, but Datanodes finalize asynchronously as they receive the finalize command from SCM. + +### New OM Requests (Read or Write) + +**Incompatible:** Any new read or write request added to the OM. New write requests will not be able to be applied by all nodes when the cluster is in a mixed version. New read requests may appear and disappear to the client as a mixed version OM Ratis group changes leaders. + +**Pattern:** Block new write requests until their feature is finalized by putting `DisallowedUntilOmVersion(OzoneManagerVersion.X)` on the `preExecute` method of its `OMClientRequest` subclass. Block new read requests until their feature is finalized by putting `DisallowedUntilOmVersion(OzoneManagerVersion.X)` on helper method the read delegates to in `OzoneManagerRequestHandler`. + +### New Fields or Behavior in Existing OM Requests (Read or Write) + +**Incompatible:** Any new option that can be passed in by a client that changes the result of an existing read or write request. For example, adding a new replication type that blocks can be allocated with. + +**Pattern:** Intercept the request before the leader OM processes it using a `RequestFeatureValidator`. This is typically it is placed in the `OMClientRequest` subclass for write requests and in `OzoneManagerRequestHandler` for read requests. + +```java + @RequestFeatureValidator( + conditions = ValidationCondition.CLUSTER_NEEDS_FINALIZATION, + processingPhase = RequestProcessingPhase.PRE_PROCESS, + requestType = Type.SetBucketProperty + ) + public static OMRequest disallowSetBucketPropertyWithECReplicationConfig( + OMRequest req, ValidationContext ctx) throws OMException { + if (!ctx.versionManager().isAllowed(OMLayoutFeature.ERASURE_CODED_STORAGE_SUPPORT)) { Review Comment: Good catch, will add a callout for this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
