Wei-Chiu Chuang created HDDS-17653:
--------------------------------------
Summary: SCM upgrade finalization stalls when
ozone.replication.type is EC
Key: HDDS-17653
URL: https://issues.apache.org/jira/browse/HDDS-17653
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Affects Versions: 2.2.2
Reporter: Wei-Chiu Chuang
SCM finalization waits for an OPEN RATIS/THREE pipeline, but with
{{ozone.replication.type=EC}} its background pipeline creator creates only
RATIS/ONE pipelines. Finalization remains in progress despite enough healthy
Datanodes and successful EC writes.
h3. Reproduction
# Start Ozone 2.1.2 with five Datanodes and these settings on SCM and clients:
{noformat}
ozone.replication.type=EC
ozone.replication=rs-3-2-1024k
ozone.server.default.replication.type=EC
ozone.server.default.replication=rs-3-2-1024k
{noformat}
The trial used {{ozone.scm.pipeline.creation.interval=5s}} and
{{hdds.scm.safemode.min.datanode=1}}. All five Datanodes were verified HEALTHY
before finalization.
# Create an RS-3-2 bucket, write a key, and verify its contents.
# Stop all services and replace binaries with 2.2.2, preserving configuration,
addresses, and data. Start SCM and Datanodes normally and OM with {{--upgrade}}.
# Wait for SCM to exit safe mode, verify five HEALTHY Datanodes and the
existing key, then run {{ozone admin scm finalizeupgrade}}.
# Observe the CLI remaining blocked while SCM repeatedly logs:
{noformat}
Waiting for at least one open Ratis 3 pipeline after SCM finalization.
{noformat}
h3. Observed result
SCM advanced its layout and reached MLV_EQUALS_SLV, but reported
FINALIZATION_IN_PROGRESS. There were five OPEN RATIS/ONE pipelines and an OPEN
EC pipeline, with no RATIS/THREE pipeline. A new EC key could be written, read
back, and compared successfully while finalization remained blocked. The
observation captured 27 repeated wait messages over more than two minutes
without automatic completion.
Causal control: with the EC configuration unchanged, run:
{noformat}
ozone admin pipeline create --replication-factor=THREE
{noformat}
Once that pipeline became OPEN, the original finalizeupgrade command completed
with exit code 0 and SCM reported FINALIZATION_DONE. This confirms the wait was
caused by the missing Ratis pipeline, rather than insufficient Datanodes or
failed EC allocation.
h3. Expected result
Finalization should complete for a supported EC-default configuration. Pipeline
creation and the finalization completion condition must agree, while preserving
the required layout and Datanode upgrade guarantees.
h3. Fix and validation
Make the finalization pipeline requirement consistent with background pipeline
creation for EC-default clusters. Add upgrade regression coverage with enough
healthy Datanodes, including an empty cluster and a cluster serving EC writes.
Retain Ratis-default coverage and provide actionable progress if required
pipelines cannot be created.
h3. Scope and related work
The trigger is {{ozone.replication.type=EC}} on SCM. EC buckets alone, or only
setting {{ozone.server.default.replication.type=EC}}, do not exercise this
background-creation branch. The reproduction was non-HA and non-secure; no data
loss was observed.
HDDS-17652 tracks prerequisite validation and CLI progress; HDDS-17624
documents the upgrade procedure. Review HDDS-15138 for relevant
background-pipeline changes and possible backport reuse; that commit is not in
the tested 2.2.2 tag.
h3. Source
https://github.com/apache/ozone/blob/ozone-2.2.2/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/BackgroundPipelineCreator.java#L205-L236
https://github.com/apache/ozone/blob/ozone-2.2.2/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/SCMUpgradeFinalizer.java#L173-L197
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
