Siyao Meng created HDDS-17622:
---------------------------------
Summary: RackScatter placement fails although a rack has a
writable datanode, because a new rack gets only one round of random picks
Key: HDDS-17622
URL: https://issues.apache.org/jira/browse/HDDS-17622
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-4-retry-skipped-rack-in-rackscatter-unique-rack-phase.patch,
TestBugCR4RackScatterSingleRoundOfPicks.java
h3. Mechanism
{{SCMContainerPlacementRackScatter.chooseDatanodesInternal}} first picks one
node on each rack that holds no replica yet ("unique racks"), then fills the
rest with a same-rack fallback. For each rack, {{chooseNode}} draws up to 5
random nodes ({{INNER_LOOP_MAX_RETRY}}) and excludes each one that
{{isValidNode}} rejects. {{chooseNodesFromRacks}} can retry a rack whose picks
all missed in a later round ({{skippedRacks}}, ending after 4 rounds without
progress, {{OUTER_LOOP_MAX_RETRY}} + 1), but the unique-rack phase calls it
with {{maxOuterLoopIterations = maxReplicasPerRack}}. For EC on at least as
many racks as replicas that is 1, so a rack gets a single round of 5 picks. If
those miss, the method throws {{FAILED_TO_FIND_HEALTHY_NODES}} ("Chosen nodes
size from Unique Racks: 0, but required nodes to choose from Unique Racks: 1 do
not match"), and the same-rack fallback is not reached.
Full, STALE, decommissioning and maintenance datanodes stay in the network
topology and are not in the excluded list, so on a rack where most datanodes
cannot take a replica, 5 picks often miss the ones that can.
h3. Trigger
An EC container needs a target on a new rack, most datanodes of that rack are
full or not writable, and it is the only rack without a replica, for example a
cluster with exactly as many racks as the EC width (rs-3-2 on 5 racks). With
more candidate racks, the first round moves on to the next rack, and the
placement fails only if every candidate rack misses. For example rs-3-2 with
index 1 to 4 on four racks and index 5 missing, and a fifth rack with 10
datanodes of which one is writable: 5 picks without replacement miss it with
probability 0.5. With 20 unusable datanodes and one writable it is 16/21 (about
0.76).
h3. Impact
The placement fails although a valid target exists.
{{ReplicationManagerUtil.getTargetDatanodes}} turns it into
{{FAILED_TO_FIND_SUITABLE_NODE}}, {{ECUnderReplicationHandler}} rethrows, and
the container is requeued for the next pass, which draws again. Repair still
completes, but with failure probability p it takes p/(1-p) extra passes on
average (the under-replicated queue is processed every 30 s by default): one
more pass in the first layout above and about three in the second, while the
container runs with reduced redundancy. Each failed pass also logs an ERROR
with a stack trace, raises the container's requeue count (which sorts it behind
others with the same remaining redundancy), and runs the handler's remedies for
a missing target, such as deleting an UNHEALTHY replica if the container has
one. The same applies to every caller of the EC policy, for example copies for
decommission and mis-replication, and EC pipeline creation on the write path
({{ECPipelineProvider}} calls the same {{chooseDatanodes}} with no used nodes),
where {{WritableECContainerProvider}} hides most failures by falling back to
existing open pipelines and a second allocation attempt. Not measured end to
end; only the per placement failure rate was run.
HDDS-16634 reports the same kind of probe limit in
{{SCMContainerPlacementRackAware}}, measured as a slower decommission.
h3. Reproduction
[^TestBugCR4RackScatterSingleRoundOfPicks.java] ({{hadoop-hdds/server-scm}},
target path and Maven command in the class comment) builds the example above
with a real {{NetworkTopologyImpl}} and the real policy (only {{NodeManager}}
is a mock that returns the datanodes): rack4 has 5 full, 4 STALE and 1 writable
datanode, and rack0 to rack3 each have the used node and 2 free writable nodes.
It runs 200 placements for one target through
{{ReplicationManagerUtil.getTargetDatanodes}}. On unmodified source about half
failed (102 of 200 in one run) with {{FAILED_TO_FIND_SUITABLE_NODE}}, the other
98 chose the writable datanode of rack4, and none used the free nodes of rack0
to rack3. The test passes when the defect is present. With the patch all 200
chose the writable datanode and the test fails at the assertion on the failure
count.
h3. Patch
[^CR-4-retry-skipped-rack-in-rackscatter-unique-rack-phase.patch], against
e4ae41fc4047c99af983020b4f33bd297c9d0ac5. It also applies to master at
3fc2387040f.
The unique-rack phase no longer limits {{chooseNodesFromRacks}} to
{{maxReplicasPerRack}} rounds (the same-rack fallback already passed
{{Integer.MAX_VALUE}}), so the existing skipped-rack retry runs: a rack whose
picks all missed is tried again in the next round, with the rejected nodes
still excluded, until 4 rounds bring no progress. The round limit parameter,
now unused, is removed. The per rack limit is unchanged, so no rack gets more
than {{maxReplicasPerRack}} nodes, and the final {{validateContainerPlacement}}
check still applies. A placement that succeeded at the old limit is unchanged;
only placements that used to throw get more picks. A rack with no usable node
(as in HDDS-17362) still ends in the same exception, after 20 picks instead of
5, so such a failing placement now logs up to 4 times as many INFO lines from
{{isValidNode}} and {{chooseNode}} on each pass. This is bounded and rack
scoped, and the same-rack fallback already behaves this way.
This narrows the failure, it does not remove it: a rack is now searched up to
20 picks instead of 5. Picks are made without replacement, so a rack of up to
20 datanodes is always searched completely, and the remaining failure needs a
larger rack that is almost all unusable. With 20 unusable datanodes and one
writable, 91 of 2000 placements still failed (1/21 expected), against about 76%
before. Choosing only among datanodes that can take a replica would remove it;
that is the direction of HDDS-16634 and out of scope here. Falling through to
the same-rack fallback instead of throwing, as HDDS-16629 did for
{{SCMContainerPlacementRackAware}} when no other rack has a writable node, was
not taken: here another rack does have one, and the fallback would put the
replica on a rack that already holds one, leaving the container mis-replicated.
Covered by {{chooseOnlyWritableNodeOfMostlyUnusableRack}} in the existing
{{TestSCMContainerPlacementRackScatter}}: one writable datanode among 10 on the
only new rack must be chosen in 20 of 20 placements, and when it is made STALE
too the placement still fails with {{FAILED_TO_FIND_HEALTHY_NODES}}. Without
the change it fails with the "Unique Racks: 0" exception above. With the
change, the placement policy, pipeline placement and Replication Manager
handler suites in {{hadoop-hdds/server-scm}} pass (651 tests) and checkstyle is
clean.
Found by code review of the SCM EC container replication and placement repair
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
e4ae41fc4047c99af983020b4f33bd297c9d0ac5. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-16634 (the probe limit in {{SCMContainerPlacementRackAware}}),
HDDS-17362 (RackScatter counts a rack of out-of-service datanodes as available;
the patch does not change that case beyond the extra picks described above).
The attached patch is a proposal for review. Generated with Specula (Claude
Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
