Thanks everyone for voting. The vote has closed with 5 binding +1s (4 from PMC members who did not work directly on the feature) 3 additional +1s
I am resolving some last conflicts with the latest master branch and then I will merge the feature branch. Siyao, regarding your comments which we can address after the merge 1. The client facing OMStartFinalizeUpgradeRequest has an admin check. CompleteFinalizeUpgrade is submitted by a background service and follows the same model as other internal requests submitted by background services. If there is an issue here it is not unique to the ZDU branch and should be discussed in a different channel. 2. This looks like an AI summary that is somewhat jumbled. I've separated it into three parts: First is referencing the removal of HEALTHY_READONLY, but that seems irrelevant to the issues described. Second is referring to nodes that are STALE and DEAD and do not have their software upgraded. Only all HEALTHY nodes need to be upgraded for SCM start finalization, but once SCM finalizes, it should not allow communication with nodes in the older software. However this is checked at the time of registration and STALE and DEAD nodes can heartbeat back to SCM without reregistering. This should be fixable in the general case by sending a reregister command in response to any node that heartbeats a software version less than SCM when it is finalized. Filed HDDS-16798 <https://issues.apache.org/jira/browse/HDDS-16798>. Third is referring to the output of ozone admin upgrade status which counts Datanodes as finalized if their software and apparent versions match, even if they are less than SCM's software version. This is technically the correct definition of finalized (those nodes are finalized in the old software version), but it may lead to confusing output if the command is run when just the SCM software has been upgraded. It will show SCM as unfinalized but all Datanodes as finalized. I filed HDDS-16799 <https://issues.apache.org/jira/browse/HDDS-16799> to fix this by only counting a Datanode as finalized if its software and apparent versions match each other and SCM's software version. SCM finalization is gated by allSoftwareVersionsMatchScm to ensure all Datanodes have been upgraded before SCM begins finalization. It does not use the finalized nodes counter, so this third issue is only cosmetic and not coupled to the second issue. - Ethan On Thu, Oct 8, 2026 at 1:22 AM Sadanand Shenoy <[email protected]> wrote: > +1. Thanks to everyone who worked on this feature. > > -Sadanand > > > > > On Thu, Oct 8, 2026 at 9:36 AM Siyao Meng <[email protected]> wrote: > > > +1 (binding) > > > > Thanks Ethan, Stephen, Zita, and everyone who worked on this. > > > > I found two issues that may need addressing, but I’m fine with filing > > follow-up Jiras and handling them after the merge. > > > > 1. HDDS-16025 exposed CompleteFinalizeUpgrade through client dispatch, > but > > its handler [1] bypasses admin authorization and the normal finalization > > prerequisites. IMO preExecute() should be overridden to reject external > > requests. > > > > 2. HDDS-14671 removed HEALTHY_READONLY recovery, and HDDS-15034 added the > > finalization counter. A previously registered old datanode can resume > > heartbeats without restarting after SCM finalizes and become HEALTHY > again. > > The version-rejection branch [2] only logs and returns, and the counter > [3] > > treats matching datanode apparent/software versions as finalized even > when > > both are older than SCM. Incompatible nodes should receive > > ReregisterCommand before refreshing their heartbeat timestamp, and > > finalized counts should require SCM’s software version. > > > > Thanks, > > Siyao > > > > [1] > > > > > https://github.com/apache/ozone/blob/a9a64dbed31ba1dfee9057b7c8ddd42acb948b2f/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/upgrade/OMCompleteFinalizeUpgradeRequest.java#L68 > > > > [2] > > > > > https://github.com/apache/ozone/blob/a9a64dbed31ba1dfee9057b7c8ddd42acb948b2f/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java#L806 > > > > [3] > > > > > https://github.com/apache/ozone/blob/a9a64dbed31ba1dfee9057b7c8ddd42acb948b2f/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java#L185 > > > > On Tue, Oct 6, 2026 at 3:30 PM Ethan Rose <[email protected]> wrote: > > > > > Hi all, thanks for voting. All the content is ready for merge at this > > time: > > > > > > - master has been merged into the ZDU branch with green CI at commit > > > a99fabb9dea89b1d3961bd1eccbabee13f2047f7 today > > > - Design doc and user doc have been updated based on initial > reviews. > > > - pull/11354 <https://github.com/apache/ozone/pull/11354> is merged > > > > > > Still outstanding: > > > > > > - +1s on the design document < > > https://github.com/apache/ozone/pull/9664 > > > >, > > > which should be merged with the feature branch > > > - Ideally at least one more binding PMC +1 from someone who did not > > work > > > on the feature directly. > > > - Currently we have Stephen as part of the feature, and Uma and > > Ayush > > > from outside > > > > > > We can leave the dev <https://github.com/apache/ozone-site/pull/560> > and > > > user <https://github.com/apache/ozone-site/pull/545> docs open for > > reviews > > > for a bit after the branch merges in case people want more time to > review > > > those. I especially want to call people's attention to the dev docs > since > > > those will need to be understood by everyone once ZDU is merged. > > > > > > I will keep updating the feature branch from master until we are ready > to > > > merge it in. > > > > > > Ethan > > > > > > On Mon, Oct 5, 2026 at 5:24 AM Ayush Saxena <[email protected]> > wrote: > > > > > > > +1 > > > > > > > > -Ayush > > > > > > > > > On 5 Oct 2026, at 2:43 PM, Zita Dombi <[email protected]> > wrote: > > > > > > > > > > +1 (non binding) to merge. Disclaimer: I worked on this feature > > > > > (implementation and code review) > > > > > > > > > > Zita > > > > > > > > > > Stephen O'Donnell via dev <[email protected]> ezt írta > (időpont: > > > > 2026. > > > > > okt. 5., H, 11:05): > > > > > > > > > >> Summit, > > > > >> > > > > >> I think your questions are covered in the design from this part > > > > >> > > > > >> > > > > > > > > > > https://github.com/apache/ozone/pull/9664/changes#diff-93aadee36203b3e50605117504fae31353edba5844c084e8169fde778c1c1a1cR239 > > > > >> > > > > >> Datanodes will continue to be available for writes during > finalize. > > In > > > > >> short they will operate at the lowest version in the pipeline and > > any > > > > >> blocks currently being written will be written / committed at the > > > > earlier > > > > >> version. > > > > >> > > > > >> Stephen. > > > > >> > > > > >> > > > > >> On Mon, Oct 5, 2026 at 4:13 AM Sumit Agrawal via dev < > > > > [email protected] > > > > >>> > > > > >> wrote: > > > > >> > > > > >>> +1 for merge. > > > > >>> > > > > >>> Pipeline can exist in mixed version of DN (old and new). Design > doc > > > do > > > > >> not > > > > >>> provide much clarity about the behavior: > > > > >>> - Weather write is supported in this case? allocating new blocks > > and > > > > >>> pipeline > > > > >>> - How about pipeline / block already assigned for write? Client > > will > > > > not > > > > >> be > > > > >>> aware of changes. > > > > >>> > > > > >>> Since SCM is already finalied and DN finalization is in progress > > > (which > > > > >> can > > > > >>> take 1-2 HB for few set of DNs where SCM will know). So there may > > be > > > a > > > > >>> delay and DN upgrade may also take time. > > > > >>> > > > > >>> - How about DN supporting read and write operation when > > finalization > > > is > > > > >> in > > > > >>> progress? is it blocking or request will be rejected by DNs ? > > > > >>> > > > > >>> > > > > >>> On Thu, Oct 1, 2026 at 7:46 PM Andrey Yarovoy via dev < > > > > >>> [email protected]> > > > > >>> wrote: > > > > >>> > > > > >>>> +1 for merge. > > > > >>>> > > > > >>>> On Thu, Oct 1, 2026 at 10:01 AM Uma Maheswara Rao Gangumalla < > > > > >>>> [email protected]> wrote: > > > > >>>> > > > > >>>>> Great work. Thank you Ethan and all others for working on this. > > > > >>>>> > > > > >>>>> +1 for the merge. > > > > >>>>> > > > > >>>>> Regards, > > > > >>>>> Uma > > > > >>>>> > > > > >>>>>> On Mon, Sep 28, 2026 at 4:02 PM Ethan Rose <[email protected]> > > > > wrote: > > > > >>>>>> > > > > >>>>>>> Hi Ozone devs, > > > > >>>>>>> > > > > >>>>>>> This is a vote thread to merge the zero downtime upgrade > > feature > > > > >>>>> branch ( > > > > >>>>>>> HDDS-14496-zdu) into master, which will enable zero downtime > > > > >>> upgrades > > > > >>>>>> from > > > > >>>>>>> the first release containing this feature to all future > > releases. > > > > >>>>> Please > > > > >>>>>>> familiarize yourself with the following documents when > voting. > > > All > > > > >>> docs > > > > >>>>> PRs > > > > >>>>>> remain open so we can address questions and comments as they > > arise > > > > >>>> during > > > > >>>>>> the voting process. > > > > >>>>>> > > > > >>>>>> - Design document < > https://github.com/apache/ozone/pull/9664> > > > > >>>>>> - Branch merge checklist < > > > > >>>>> https://github.com/apache/ozone-site/pull/562 > > > > >>>>>>> > > > > >>>>>> - User Docs <https://github.com/apache/ozone-site/pull/545> > > > > >>>>>> - Developer docs < > > https://github.com/apache/ozone-site/pull/560 > > > > >>> : > > > > >>>>> This > > > > >>>>>> guide covers how to handle compatibility for zero downtime > > > > >> upgrade > > > > >>>> and > > > > >>>>>> should be understood by all developers and reviewers. > > > > >>>>>> > > > > >>>>>> Summary of changes: New Requirements for Developers > > > > >>>>>> > > > > >>>>>> *Once this branch is merged, all further commits must be ZDU > > > > >>> compatible > > > > >>>>> as > > > > >>>>>> outlined in the developer guide*. > > > > >>>>>> > > > > >>>>>> The new versioning framework created on this branch provides > > tools > > > > >> to > > > > >>>>>> safely incorporate incompatible changes, but it does not > > > > >>> automatically > > > > >>>>>> resolve them. That remains the responsibility of developers > and > > > > >>>>> reviewers. > > > > >>>>>> Improved Developer Experience > > > > >>>>>> > > > > >>>>>> Each component now uses a single ComponentVersion to track all > > > > >>>>> incompatible > > > > >>>>>> changes across disk and network as outlined in the design doc. > > > > >>>> Developers > > > > >>>>>> no longer need to reason about whether their incompatible > change > > > > >>>>> requires a > > > > >>>>>> LayoutFeature, ComponentVersion, or both. As part of this > > change, > > > > >> the > > > > >>>>>> internal upgrade framework was rewritten and exposes a simpler > > API > > > > >> to > > > > >>>>>> developers. This includes strongly typed component versions > > (with > > > > >>>> integer > > > > >>>>>> conversion deferred until serialization), and an isSupportedBy > > > > >> method > > > > >>>> to > > > > >>>>>> handle all version comparisons. > > > > >>>>>> Improved Admin Experience > > > > >>>>>> > > > > >>>>>> To ensure finalization proceeds in the correct order as > outlined > > > in > > > > >>> the > > > > >>>>>> design document, admins no longer have to finalize OM and SCM > > > > >>>>> separately. A > > > > >>>>>> single ozone admin upgrade finalize command triggers > > asynchronous > > > > >>>>>> finalization throughout the cluster in the defined order. A > > single > > > > >>>> status > > > > >>>>>> endpoint can be queried by ozone admin upgrade status, and > > clients > > > > >>> can > > > > >>>>>> trigger and block on finalization with one ozone admin upgrade > > > > >>> finalize > > > > >>>>>> --wait command, which handles polling of the status endpoint > by > > > the > > > > >>>>> client > > > > >>>>>> automatically and is idempotent. > > > > >>>>>> > > > > >>>>>> Additionally, a Grafana dashboard has been added to provide a > > > heads > > > > >>> up > > > > >>>>> view > > > > >>>>>> of all components during an upgrade. It includes filtering to > > zoom > > > > >> in > > > > >>>> on > > > > >>>>> a > > > > >>>>>> particular area and aggregates across Datanodes to handle > large > > > > >>>> clusters. > > > > >>>>>> Removed Prepare For Upgrade > > > > >>>>>> > > > > >>>>>> The "prepare for upgrade" command which put OMs in a read-only > > > mode > > > > >>>>> before > > > > >>>>>> an upgrade is no longer required. The CLI has been left as a > > no-op > > > > >>> for > > > > >>>>>> compatibility with older upgrade scripts. See the developer > > guide > > > > >>>> linked > > > > >>>>>> above for instructions to handle incompatible changes to OM > > write > > > > >>>>> requests. > > > > >>>>>> Work In Progress > > > > >>>>>> > > > > >>>>>> There is some ongoing work we will continue in parallel with > the > > > > >>> merge > > > > >>>>> vote > > > > >>>>>> and finish before the branch is merged: > > > > >>>>>> > > > > >>>>>> - We are currently merging the latest master branch into the > > > > >>> feature > > > > >>>>>> branch, resolving conflicts, and running it through CI. The > > > > >> final > > > > >>>> hash > > > > >>>>>> for > > > > >>>>>> merge will be shared here when ready. > > > > >>>>>> - The ZDU design doc will be updated based on the latest > > copilot > > > > >>>>> review > > > > >>>>>> and other minor deviations identified from the resulting > > > > >>>>> implementation. > > > > >>>>>> - A PR to add a missed admin check on the ozone admin > upgrade > > > > >>> status > > > > >>>>>> command is in flight: > > > > >> https://github.com/apache/ozone/pull/11354 > > > > >>>>>> > > > > >>>>>> Future Work > > > > >>>>>> > > > > >>>>>> As mentioned in the merge checklist, the current OM request > > > > >>> versioning > > > > >>>>>> framework on master was left intact on the ZDU branch. > However, > > > the > > > > >>>>> opt-in > > > > >>>>>> annotation based approach does not suit the new ZDU > requirements > > > > >>> where > > > > >>>>>> every new request needs to be versioned. Dev work has started > > on a > > > > >>> new > > > > >>>>>> framework, but the change is large and was deliberately saved > > for > > > > >>>> master > > > > >>>>>> after the branch merge so that it can be reviewed by a wider > > > > >>> audience. > > > > >>>>>> > > > > >>>>>> Additionally, we will be investigating static analysis and AI > > > > >> skills > > > > >>> to > > > > >>>>>> flag potentially incompatible changes during code reviews and > > the > > > > >>>> release > > > > >>>>>> process. > > > > >>>>>> ------------------------------ > > > > >>>>>> > > > > >>>>>> Thanks to Stephen, Zita, and Roland who also worked on the > > > > >>> development > > > > >>>> of > > > > >>>>>> this feature and to everyone who shared inputs on the design. > > > > >>>>>> > > > > >>>>>> We will leave the vote thread open for at least 7 days, > although > > > > >> more > > > > >>>>> time > > > > >>>>>> may be required for developers to familiarize themselves with > > the > > > > >> new > > > > >>>> ZDU > > > > >>>>>> requirements. > > > > >>>>>> > > > > >>>>>> - Ethan > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>>> > > > > >>>> -- > > > > >>>> Thanks, > > > > >>>> Andrey. > > > > >>>> > > > > >>> > > > > >>> > > > > >>> -- > > > > >>> *Sumit Agrawal* | Senior Staff Engineer > > > > >>> cloudera.com <https://www.cloudera.com> > > > > >>> [image: Cloudera] <https://www.cloudera.com/> > > > > >>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> > > [image: > > > > >>> Cloudera on Facebook] <https://www.facebook.com/cloudera> > [image: > > > > >> Cloudera > > > > >>> on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > >>> ------------------------------ > > > > >>> > > > > >> > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > >
