Hi Ethan, 1. That’s fair. I filed HDDS-16813 to track this separately. 2. Thanks for filing the Jiras.
-Siyao On Thu, Oct 8, 2026 at 3:43 PM Ethan Rose <[email protected]> wrote: > Thanks everyone for voting. The vote has closed with > 5 binding +1s (4 from PMC members who did not work directly on the feature) > 3 additional +1s > > I am resolving some last conflicts with the latest master branch and then I > will merge the feature branch. > > Siyao, regarding your comments which we can address after the merge > > 1. > > The client facing OMStartFinalizeUpgradeRequest has an admin check. > CompleteFinalizeUpgrade is submitted by a background service and follows > the same model as other internal requests submitted by background > services. > If there is an issue here it is not unique to the ZDU branch and should > be > discussed in a different channel. > 2. > > This looks like an AI summary that is somewhat jumbled. I've separated > it into three parts: > > First is referencing the removal of HEALTHY_READONLY, but that seems > irrelevant to the issues described. > > Second is referring to nodes that are STALE and DEAD and do not have their > software upgraded. Only all HEALTHY nodes need to be upgraded for SCM start > finalization, but once SCM finalizes, it should not allow communication > with nodes in the older software. However this is checked at the time of > registration and STALE and DEAD nodes can heartbeat back to SCM without > reregistering. This should be fixable in the general case by sending a > reregister command in response to any node that heartbeats a software > version less than SCM when it is finalized. Filed HDDS-16798 > <https://issues.apache.org/jira/browse/HDDS-16798>. > > Third is referring to the output of ozone admin upgrade status which counts > Datanodes as finalized if their software and apparent versions match, even > if they are less than SCM's software version. This is technically the > correct definition of finalized (those nodes are finalized in the old > software version), but it may lead to confusing output if the command is > run when just the SCM software has been upgraded. It will show SCM as > unfinalized but all Datanodes as finalized. I filed HDDS-16799 > <https://issues.apache.org/jira/browse/HDDS-16799> to fix this by only > counting a Datanode as finalized if its software and apparent versions > match each other and SCM's software version. > > SCM finalization is gated by allSoftwareVersionsMatchScm to ensure all > Datanodes have been upgraded before SCM begins finalization. It does not > use the finalized nodes counter, so this third issue is only cosmetic and > not coupled to the second issue. > > - Ethan > > On Thu, Oct 8, 2026 at 1:22 AM Sadanand Shenoy <[email protected]> wrote: > > > +1. Thanks to everyone who worked on this feature. > > > > -Sadanand > > > > > > > > > > On Thu, Oct 8, 2026 at 9:36 AM Siyao Meng <[email protected]> wrote: > > > > > +1 (binding) > > > > > > Thanks Ethan, Stephen, Zita, and everyone who worked on this. > > > > > > I found two issues that may need addressing, but I’m fine with filing > > > follow-up Jiras and handling them after the merge. > > > > > > 1. HDDS-16025 exposed CompleteFinalizeUpgrade through client dispatch, > > but > > > its handler [1] bypasses admin authorization and the normal > finalization > > > prerequisites. IMO preExecute() should be overridden to reject external > > > requests. > > > > > > 2. HDDS-14671 removed HEALTHY_READONLY recovery, and HDDS-15034 added > the > > > finalization counter. A previously registered old datanode can resume > > > heartbeats without restarting after SCM finalizes and become HEALTHY > > again. > > > The version-rejection branch [2] only logs and returns, and the counter > > [3] > > > treats matching datanode apparent/software versions as finalized even > > when > > > both are older than SCM. Incompatible nodes should receive > > > ReregisterCommand before refreshing their heartbeat timestamp, and > > > finalized counts should require SCM’s software version. > > > > > > Thanks, > > > Siyao > > > > > > [1] > > > > > > > > > https://github.com/apache/ozone/blob/a9a64dbed31ba1dfee9057b7c8ddd42acb948b2f/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/upgrade/OMCompleteFinalizeUpgradeRequest.java#L68 > > > > > > [2] > > > > > > > > > https://github.com/apache/ozone/blob/a9a64dbed31ba1dfee9057b7c8ddd42acb948b2f/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java#L806 > > > > > > [3] > > > > > > > > > https://github.com/apache/ozone/blob/a9a64dbed31ba1dfee9057b7c8ddd42acb948b2f/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java#L185 > > > > > > On Tue, Oct 6, 2026 at 3:30 PM Ethan Rose <[email protected]> wrote: > > > > > > > Hi all, thanks for voting. All the content is ready for merge at this > > > time: > > > > > > > > - master has been merged into the ZDU branch with green CI at > commit > > > > a99fabb9dea89b1d3961bd1eccbabee13f2047f7 today > > > > - Design doc and user doc have been updated based on initial > > reviews. > > > > - pull/11354 <https://github.com/apache/ozone/pull/11354> is > merged > > > > > > > > Still outstanding: > > > > > > > > - +1s on the design document < > > > https://github.com/apache/ozone/pull/9664 > > > > >, > > > > which should be merged with the feature branch > > > > - Ideally at least one more binding PMC +1 from someone who did > not > > > work > > > > on the feature directly. > > > > - Currently we have Stephen as part of the feature, and Uma and > > > Ayush > > > > from outside > > > > > > > > We can leave the dev <https://github.com/apache/ozone-site/pull/560> > > and > > > > user <https://github.com/apache/ozone-site/pull/545> docs open for > > > reviews > > > > for a bit after the branch merges in case people want more time to > > review > > > > those. I especially want to call people's attention to the dev docs > > since > > > > those will need to be understood by everyone once ZDU is merged. > > > > > > > > I will keep updating the feature branch from master until we are > ready > > to > > > > merge it in. > > > > > > > > Ethan > > > > > > > > On Mon, Oct 5, 2026 at 5:24 AM Ayush Saxena <[email protected]> > > wrote: > > > > > > > > > +1 > > > > > > > > > > -Ayush > > > > > > > > > > > On 5 Oct 2026, at 2:43 PM, Zita Dombi <[email protected]> > > wrote: > > > > > > > > > > > > +1 (non binding) to merge. Disclaimer: I worked on this feature > > > > > > (implementation and code review) > > > > > > > > > > > > Zita > > > > > > > > > > > > Stephen O'Donnell via dev <[email protected]> ezt írta > > (időpont: > > > > > 2026. > > > > > > okt. 5., H, 11:05): > > > > > > > > > > > >> Summit, > > > > > >> > > > > > >> I think your questions are covered in the design from this part > > > > > >> > > > > > >> > > > > > > > > > > > > > > > https://github.com/apache/ozone/pull/9664/changes#diff-93aadee36203b3e50605117504fae31353edba5844c084e8169fde778c1c1a1cR239 > > > > > >> > > > > > >> Datanodes will continue to be available for writes during > > finalize. > > > In > > > > > >> short they will operate at the lowest version in the pipeline > and > > > any > > > > > >> blocks currently being written will be written / committed at > the > > > > > earlier > > > > > >> version. > > > > > >> > > > > > >> Stephen. > > > > > >> > > > > > >> > > > > > >> On Mon, Oct 5, 2026 at 4:13 AM Sumit Agrawal via dev < > > > > > [email protected] > > > > > >>> > > > > > >> wrote: > > > > > >> > > > > > >>> +1 for merge. > > > > > >>> > > > > > >>> Pipeline can exist in mixed version of DN (old and new). Design > > doc > > > > do > > > > > >> not > > > > > >>> provide much clarity about the behavior: > > > > > >>> - Weather write is supported in this case? allocating new > blocks > > > and > > > > > >>> pipeline > > > > > >>> - How about pipeline / block already assigned for write? Client > > > will > > > > > not > > > > > >> be > > > > > >>> aware of changes. > > > > > >>> > > > > > >>> Since SCM is already finalied and DN finalization is in > progress > > > > (which > > > > > >> can > > > > > >>> take 1-2 HB for few set of DNs where SCM will know). So there > may > > > be > > > > a > > > > > >>> delay and DN upgrade may also take time. > > > > > >>> > > > > > >>> - How about DN supporting read and write operation when > > > finalization > > > > is > > > > > >> in > > > > > >>> progress? is it blocking or request will be rejected by DNs ? > > > > > >>> > > > > > >>> > > > > > >>> On Thu, Oct 1, 2026 at 7:46 PM Andrey Yarovoy via dev < > > > > > >>> [email protected]> > > > > > >>> wrote: > > > > > >>> > > > > > >>>> +1 for merge. > > > > > >>>> > > > > > >>>> On Thu, Oct 1, 2026 at 10:01 AM Uma Maheswara Rao Gangumalla < > > > > > >>>> [email protected]> wrote: > > > > > >>>> > > > > > >>>>> Great work. Thank you Ethan and all others for working on > this. > > > > > >>>>> > > > > > >>>>> +1 for the merge. > > > > > >>>>> > > > > > >>>>> Regards, > > > > > >>>>> Uma > > > > > >>>>> > > > > > >>>>>> On Mon, Sep 28, 2026 at 4:02 PM Ethan Rose < > [email protected]> > > > > > wrote: > > > > > >>>>>> > > > > > >>>>>>> Hi Ozone devs, > > > > > >>>>>>> > > > > > >>>>>>> This is a vote thread to merge the zero downtime upgrade > > > feature > > > > > >>>>> branch ( > > > > > >>>>>>> HDDS-14496-zdu) into master, which will enable zero > downtime > > > > > >>> upgrades > > > > > >>>>>> from > > > > > >>>>>>> the first release containing this feature to all future > > > releases. > > > > > >>>>> Please > > > > > >>>>>>> familiarize yourself with the following documents when > > voting. > > > > All > > > > > >>> docs > > > > > >>>>> PRs > > > > > >>>>>> remain open so we can address questions and comments as they > > > arise > > > > > >>>> during > > > > > >>>>>> the voting process. > > > > > >>>>>> > > > > > >>>>>> - Design document < > > https://github.com/apache/ozone/pull/9664> > > > > > >>>>>> - Branch merge checklist < > > > > > >>>>> https://github.com/apache/ozone-site/pull/562 > > > > > >>>>>>> > > > > > >>>>>> - User Docs < > https://github.com/apache/ozone-site/pull/545> > > > > > >>>>>> - Developer docs < > > > https://github.com/apache/ozone-site/pull/560 > > > > > >>> : > > > > > >>>>> This > > > > > >>>>>> guide covers how to handle compatibility for zero downtime > > > > > >> upgrade > > > > > >>>> and > > > > > >>>>>> should be understood by all developers and reviewers. > > > > > >>>>>> > > > > > >>>>>> Summary of changes: New Requirements for Developers > > > > > >>>>>> > > > > > >>>>>> *Once this branch is merged, all further commits must be ZDU > > > > > >>> compatible > > > > > >>>>> as > > > > > >>>>>> outlined in the developer guide*. > > > > > >>>>>> > > > > > >>>>>> The new versioning framework created on this branch provides > > > tools > > > > > >> to > > > > > >>>>>> safely incorporate incompatible changes, but it does not > > > > > >>> automatically > > > > > >>>>>> resolve them. That remains the responsibility of developers > > and > > > > > >>>>> reviewers. > > > > > >>>>>> Improved Developer Experience > > > > > >>>>>> > > > > > >>>>>> Each component now uses a single ComponentVersion to track > all > > > > > >>>>> incompatible > > > > > >>>>>> changes across disk and network as outlined in the design > doc. > > > > > >>>> Developers > > > > > >>>>>> no longer need to reason about whether their incompatible > > change > > > > > >>>>> requires a > > > > > >>>>>> LayoutFeature, ComponentVersion, or both. As part of this > > > change, > > > > > >> the > > > > > >>>>>> internal upgrade framework was rewritten and exposes a > simpler > > > API > > > > > >> to > > > > > >>>>>> developers. This includes strongly typed component versions > > > (with > > > > > >>>> integer > > > > > >>>>>> conversion deferred until serialization), and an > isSupportedBy > > > > > >> method > > > > > >>>> to > > > > > >>>>>> handle all version comparisons. > > > > > >>>>>> Improved Admin Experience > > > > > >>>>>> > > > > > >>>>>> To ensure finalization proceeds in the correct order as > > outlined > > > > in > > > > > >>> the > > > > > >>>>>> design document, admins no longer have to finalize OM and > SCM > > > > > >>>>> separately. A > > > > > >>>>>> single ozone admin upgrade finalize command triggers > > > asynchronous > > > > > >>>>>> finalization throughout the cluster in the defined order. A > > > single > > > > > >>>> status > > > > > >>>>>> endpoint can be queried by ozone admin upgrade status, and > > > clients > > > > > >>> can > > > > > >>>>>> trigger and block on finalization with one ozone admin > upgrade > > > > > >>> finalize > > > > > >>>>>> --wait command, which handles polling of the status endpoint > > by > > > > the > > > > > >>>>> client > > > > > >>>>>> automatically and is idempotent. > > > > > >>>>>> > > > > > >>>>>> Additionally, a Grafana dashboard has been added to provide > a > > > > heads > > > > > >>> up > > > > > >>>>> view > > > > > >>>>>> of all components during an upgrade. It includes filtering > to > > > zoom > > > > > >> in > > > > > >>>> on > > > > > >>>>> a > > > > > >>>>>> particular area and aggregates across Datanodes to handle > > large > > > > > >>>> clusters. > > > > > >>>>>> Removed Prepare For Upgrade > > > > > >>>>>> > > > > > >>>>>> The "prepare for upgrade" command which put OMs in a > read-only > > > > mode > > > > > >>>>> before > > > > > >>>>>> an upgrade is no longer required. The CLI has been left as a > > > no-op > > > > > >>> for > > > > > >>>>>> compatibility with older upgrade scripts. See the developer > > > guide > > > > > >>>> linked > > > > > >>>>>> above for instructions to handle incompatible changes to OM > > > write > > > > > >>>>> requests. > > > > > >>>>>> Work In Progress > > > > > >>>>>> > > > > > >>>>>> There is some ongoing work we will continue in parallel with > > the > > > > > >>> merge > > > > > >>>>> vote > > > > > >>>>>> and finish before the branch is merged: > > > > > >>>>>> > > > > > >>>>>> - We are currently merging the latest master branch into > the > > > > > >>> feature > > > > > >>>>>> branch, resolving conflicts, and running it through CI. > The > > > > > >> final > > > > > >>>> hash > > > > > >>>>>> for > > > > > >>>>>> merge will be shared here when ready. > > > > > >>>>>> - The ZDU design doc will be updated based on the latest > > > copilot > > > > > >>>>> review > > > > > >>>>>> and other minor deviations identified from the resulting > > > > > >>>>> implementation. > > > > > >>>>>> - A PR to add a missed admin check on the ozone admin > > upgrade > > > > > >>> status > > > > > >>>>>> command is in flight: > > > > > >> https://github.com/apache/ozone/pull/11354 > > > > > >>>>>> > > > > > >>>>>> Future Work > > > > > >>>>>> > > > > > >>>>>> As mentioned in the merge checklist, the current OM request > > > > > >>> versioning > > > > > >>>>>> framework on master was left intact on the ZDU branch. > > However, > > > > the > > > > > >>>>> opt-in > > > > > >>>>>> annotation based approach does not suit the new ZDU > > requirements > > > > > >>> where > > > > > >>>>>> every new request needs to be versioned. Dev work has > started > > > on a > > > > > >>> new > > > > > >>>>>> framework, but the change is large and was deliberately > saved > > > for > > > > > >>>> master > > > > > >>>>>> after the branch merge so that it can be reviewed by a wider > > > > > >>> audience. > > > > > >>>>>> > > > > > >>>>>> Additionally, we will be investigating static analysis and > AI > > > > > >> skills > > > > > >>> to > > > > > >>>>>> flag potentially incompatible changes during code reviews > and > > > the > > > > > >>>> release > > > > > >>>>>> process. > > > > > >>>>>> ------------------------------ > > > > > >>>>>> > > > > > >>>>>> Thanks to Stephen, Zita, and Roland who also worked on the > > > > > >>> development > > > > > >>>> of > > > > > >>>>>> this feature and to everyone who shared inputs on the > design. > > > > > >>>>>> > > > > > >>>>>> We will leave the vote thread open for at least 7 days, > > although > > > > > >> more > > > > > >>>>> time > > > > > >>>>>> may be required for developers to familiarize themselves > with > > > the > > > > > >> new > > > > > >>>> ZDU > > > > > >>>>>> requirements. > > > > > >>>>>> > > > > > >>>>>> - Ethan > > > > > >>>>>> > > > > > >>>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> -- > > > > > >>>> Thanks, > > > > > >>>> Andrey. > > > > > >>>> > > > > > >>> > > > > > >>> > > > > > >>> -- > > > > > >>> *Sumit Agrawal* | Senior Staff Engineer > > > > > >>> cloudera.com <https://www.cloudera.com> > > > > > >>> [image: Cloudera] <https://www.cloudera.com/> > > > > > >>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> > > > [image: > > > > > >>> Cloudera on Facebook] <https://www.facebook.com/cloudera> > > [image: > > > > > >> Cloudera > > > > > >>> on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > > >>> ------------------------------ > > > > > >>> > > > > > >> > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > >
