errose28 opened a new pull request, #11956:
URL: https://github.com/apache/ozone/pull/11956
## What changes were proposed in this pull request?
- **DRAFT** still needs some cleanup for the tests and the metric. EC
pipeline safemode rule must be added in a dependent change
- **Split from #11861** to handle the write path separately. Some tests from
that PR were adapted for this one.
- Generated by Claude Opus 5.5 and pending more human review from me.
SCM could choose open pipelines for new writes even when some of their
datanodes had not registered. SCM loads open pipelines from its DB into
`PipelineStateManager`, together with the datanodes they had when they were
saved. Nothing on the write path checked whether those datanodes had registered
since SCM started. The pipeline scrubber would not close these pipelines until
5 minutes after safemode exit.
This PR adds `PipelineManager#getWritablePipelines` for use on the write
path. It returns only pipelines whose datanodes are all registered with SCM.
Existing `PipelineManager` methods for reading pipelines remain intact since we
need all the pipelines for things like closing pipelines, listing pipelines for
the CLI, and computing safemode exit criteria.
The PR also adds the SCM metric `NumOpenPipelinesWithUnregisteredNodes` to
`SCMPipelineMetrics`. It is a gauge of the open pipelines that have at least
one unregistered datanode, which are the pipelines hidden from writes and
waiting for the scrubber. It is recalculated each time metrics are collected
rather than updated by the scrubber, so it is accurate even before the scrubber
starts. Like the other pipeline metrics, it is a single total across Ratis and
EC.
### Resulting behavior
- **Pipeline limits:** Pipelines with unregistered datanodes still count
toward the pipeline limits. The limit in `RatisPipelineProvider` reads
`PipelineStateManager`, so the background pipeline creator does not create
pipelines past the limit while these pipelines take up slots. The background
creator never creates EC pipelines; `WritableECContainerProvider` creates them
on demand during writes, and its limit uses `getPipelineCount`, which is not
filtered. Once the scrubber closes a pipeline, its slot becomes free: the
background creator refills Ratis slots, and the next EC write that needs a
pipeline refills EC slots.
- **Handling unregistered pipelines on write before the scrubber runs:** We
need to ensure that piplines with unregistered datanodes do not fill the
pipeline limit for EC or Ratis, which would prevent all writes until the
scrubber runs.
- For Ratis, healthy pipeline safemode rule guarantees a minimum threshold
of write pipelines with all nodes registered are available. We will always have
at least this number of usable pipelines at safemode exit, until
`hdds.scm.wait.time.after.safemode.exit` plus
one`ozone.scm.pipeline.scrub.interval` (about 7.5 minutes by default) elapses.
After this point, the pipelines with unregistered nodes will be closed and we
can refill the pipeline limit with viable pipelines.
- There is currently no EC pipeline rule for safemode exit, but we need to
add one to provide the same guarantee. This would check node registration
against stored SCM pipelines since EC pipelines are not reported by datanodes.
- **Registration window:** Datanodes have until the scrubber's first run to
register: (time spent in safemode) plus
`hdds.scm.wait.time.after.safemode.exit` (default 5m) plus up to one
`ozone.scm.pipeline.scrub.interval`
(default 150s). If they miss it, the scrubber closes their pipelines.
- **Cost:** Each `getWritablePipelines` call does one `isNodeRegistered`
lookup per datanode of each matching pipeline, and so does each collection of
the new gauge. Each lookup takes the
`NodeStateMap` read lock, which readers share.
## What is the link to the Apache JIRA
HDDS-17580
## How was this patch tested?
- Unit tests added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
