Wei-Chiu Chuang created HDDS-17652:
--------------------------------------
Summary: Validate pipeline prerequisites before SCM upgrade
finalization
Key: HDDS-17652
URL: https://issues.apache.org/jira/browse/HDDS-17652
Project: Apache Ozone
Issue Type: Improvement
Reporter: Wei-Chiu Chuang
SCM upgrade finalization can advance its layout version and then wait for a
RATIS/THREE pipeline that the current cluster cannot form. The CLI gives no
useful progress while this prerequisite remains unmet.
Observed during a non-HA, non-secure Docker upgrade from Ozone 2.1.2 to 2.2.2:
# Start with one healthy Datanode and default replication ONE.
# Upgrade all services to 2.2.2, keeping their data and configuration. Verify
reads and writes while pre-finalized.
# Run {{ozone admin scm finalizeupgrade}}.
SCM finalized the STORAGE_SPACE_DISTRIBUTION layout feature and reached
MLV_EQUALS_SLV, then repeatedly logged:
{noformat}
Waiting for at least one open Ratis 3 pipeline after SCM finalization.
{noformat}
The CLI remained waiting. After two additional Datanodes joined and a
RATIS/THREE pipeline opened, finalization completed successfully. No data loss
was observed.
Requested improvement:
* Before making layout changes, check whether the cluster has enough eligible
Datanodes to form the required pipeline. If not, fail with an actionable
message explaining the requirement and preserve the pre-finalized state.
* If pipeline creation is delayed after finalization starts, report the waiting
reason and current progress to the operator. Do not imply that a timeout rolls
back completed layout changes.
* Add regression coverage for insufficient Datanodes and successful
finalization once prerequisites are satisfied.
This is a prerequisite-validation and operator-feedback improvement; it does
not propose changing the required replication factor or claim a data-loss
defect.
Found while validating HDDS-17624. The documentation change describes the
prerequisite:
https://github.com/apache/ozone-site/pull/594
Relevant code: SCMUpgradeFinalizer waits for an OPEN RATIS/THREE pipeline after
layout finalization.
https://github.com/apache/ozone/blob/ozone-2.2.2/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/SCMUpgradeFinalizer.java
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
