errose28 commented on code in PR #560: URL: https://github.com/apache/ozone-site/pull/560#discussion_r4404558193
########## docs/08-developer-guide/01-develop/05-upgrade-compatibility.md: ########## @@ -0,0 +1,355 @@ +--- +sidebar_label: Upgrade Compatibility +--- + +# Developing for Compatibility with Zero Downtime Upgrade + +Zero Downtime Upgrade (ZDU) lets the internal components of an Ozone cluster (OM, SCM, Datanode, S3 Gateway, and Recon) run in mixed versions during an upgrade with no interruption to service. Additionally, Ozone provides a window where a cluster can be downgraded after the upgrade. Both of these features require every release to uphold certain compatibility guarantees, otherwise an upgrade can silently introduce behavior like diverging OM state machines, corrupting a container replica, or leaving a Datanode unable to re-register. This page is a practical guide to what counts as an incompatible change inside the cluster, and how to safely onboard such changes to each area of the code. + +The rationale behind the framework lives in the [ZDU design document](https://github.com/apache/ozone/blob/master/hadoop-hdds/docs/content/design/zdu-design.md). Read it if you want the full model. This page is only focused on day-to-day development. + +:::note Scope +This page covers compatibility **between internal Ozone components** during an upgrade. External client ↔ Ozone-server compatibility is a separate concern. +::: + +## Background + +### Vocabulary + +- **Apparent version**: The version a component is *acting as*. It is persisted to disk and determines both the API the component exposes and the on-disk format it writes. + +- **Software version**: The highest component version contained in the running code. This is fixed by the bits that are installed. + +- **Pre-finalized**: The state a component is in when its apparent version on disk is *less than* its software version. All old features work; new features are blocked; downgrade is allowed. Components enter this state when they are first started after an upgrade, and remain here until they are either downgraded or finalized by the admin. + +- **Finalized**: The state a component reaches when its apparent version *equals* its software version. All features are allowed; downgrade is no longer possible. Components enter this state after they have been upgraded to a pre-finalized state and then given an explicit finalize command by the admin. This transition happens while the cluster is running and the cluster is expected to remain operational during finalization. + +Any behavior incompatible with an old version's disk layout or API must be mapped to a version and disabled until its corresponding version is finalized. + +### Invariants Provided to the Developer + +To maintain consistency during zero downtime upgrade, the following invariants can be relied on by developers: + +1. **For internal client/server relationships, the server is always upgraded before its client.** + - The only exception is Recon, which is a client of OM but is upgraded ahead of it. + - This is maintained by the admin following the documented component upgrade order. +2. **All internal components are running the newest software before finalization begins.** + - Ozone's upgrade framework provides best-effort enforcement of this invariant, but due to corner cases it ultimately falls on the admin to correctly execute. +3. **For internal client/server relationships, the server is always finalized before its client.** + - Again, Recon is the only exception, since it finalizes before OM. + - This is completely enforced by the upgrade framework within Ozone. + +#### Upgrade Order + +Components are upgraded in the following order to meet the client/server version invariant: + +- SCM +- Recon +- Datanodes +- OM +- S3 and HttpFS Gateways (can be upgraded simultaneously) + +Feature developers do not need to account for upgrade orders that deviate from this when handling compatibility for a feature. An out of order upgrade is considered user error. + +#### Finalization Order + +Components finalize in the same order they are upgraded. Finalization is triggered by a single admin command and the cluster ensures the correct finalization order internally. + +- SCM: Finalized via Ratis +- Datanodes: Finalized asynchronously by SCM after it finalizes itself. +- OM: Finalized via Ratis only once HDDS (SCM and all live Datanodes) have finalized + +Stateless gateway components do not need to finalize since they have no disk state to manage on downgrade and do not communicate with each other. Their compatibility with the rest of the cluster is handled solely by the guarantee that the server they call remains backwards compatible with older client versions. Recon currently finalizes immediately on upgrade, although downgrade support for Recon may be added in the future. + +## Quick Start + +Within this framework, most incompatible changes can be handled by upholding two rules during feature development: + +1. **Components with the same apparent version expose the same API surface and persist data in the same format.** +2. **For internal client/server relationships, the server remains backwards compatible with older clients.** + +When a change to API surface or disk layout must be introduced, a new component version must be added to gate the feature. To do this, first add an entry to the affected components' version enums: + +- [`OzoneManagerVersion.java`](https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneManagerVersion.java) — used within the OM ring and reported to external clients. +- [`HDDSVersion.java`](https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HDDSVersion.java) — shared by SCM and Datanodes so SCM can orchestrate Datanode finalization and shared with clients. +- [`ReconVersion.java`](https://github.com/apache/ozone/blob/master/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/upgrade/ReconVersion.java) — Recon's own internal disk-format version which is currently not shared with other components. + +:::warning +Do **not** add new entries to `OMLayoutFeature` or `HDDSLayoutFeature`. Those enums are frozen after the migration to ZDU's new versioning framework. Every new version goes into `OzoneManagerVersion`, `HDDSVersion`, or `ReconVersion`. +::: + +```java +public enum OzoneManagerVersion implements ComponentVersion { + // Previous versions... + MY_NEW_FEATURE(101 /*or next available version integer*/, "My new feature"), +} +``` + +Most incompatible changes are handled by gating the new behavior on the component's apparent version through its version manager: + +```java +if (ozoneManager.getVersionManager().isAllowed(OzoneManagerVersion.MY_NEW_FEATURE)) { + // All capabilities of MY_NEW_FEATURE can be used, even if they are incompatible with older components. +} else { + // Must continue to act as the older version without using incompatible changes from MY_NEW_FEATURE. +} +``` + +`isAllowed(...)` returns true only once the component's apparent version is at least the version you name. Each component has a version manager with an equivalent method. + +:::warning +It is incorrect to write the following: + +```java +if (!ozoneManager.getVersionManager().needsFinalization()) { + // Use MY_NEW_FEATURE +} else { + // Bypass MY_NEW_FEATURE +} +``` + +New versions will continue be added in subsequent releases after `MY_NEW_FEATURE`. Say `MY_NEW_FEATURE_2` is added in the next release. When upgrading from software version `MY_NEW_FEATURE` to `MY_NEW_FEATURE_2`, `MY_NEW_FEATURE` must remain finalized and usable from the previous upgrade even if the component is not yet finalized for `MY_NEW_FEATURE_2`. +::: + +## Committing Changes + +Ozone's component versions are a logical versioning framework decoupled from official Ozone releases. The system is expected to work correctly regardless of which commit a release is cut from. To maintain this property, changes requiring versioning must be checked in with the same commit that the version is introduced. Splitting this over multiple commits will create an incompatiblity if a release is cut between the commits: + +1. **Incompatible: Adding an incompatible change and then adding a version gate for it in a follow-up commit.** A release cut between these two commits will release the incompatible change without the version gate. + +2. **Incompatible: Adding a version in one commit and using it in a follow-up commit.** A release cut between these two commits will finalize the version as a no-op. The next release will then start using the feature automatically as if it was introduced in the previous version. + +3. **Incompatible: Extending the usage of an existing version in follow-up commits.** This breaks for the same reason as (2). A new version must be used for the new functionality. + +One large feature may introduce incompatible changes in multiple areas which take multiple commits to add, however it is easier to reason about these if they are grouped by one apparent version. There are two strategies developers can use for this: + +1. **Build these features on feature branches.** Feature branches are not release candidates, so they can contain incompatible changes between commits. A single version just needs to be added to gate all incompatible behavior before the branch is merged back to master. + +2. **Place the incompatible changes behind a disabled feature flag, and switch this flag to a version gate when the feature is ready to be released.** If multiple incompatible changes for a feature are being added to master, they can each be gated with the same pattern while under development: Review Comment: In most cases the flag will need to be a config key so it is reachable from everywhere it is needed, including acceptance tests. I added some guidance for naming and describing these keys instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
