[
https://issues.apache.org/jira/browse/HDDS-16871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-16871:
-------------------------------------
Assignee: Andrey Yarovoy
> EC: Share failed datanodes across the blocks of a key read
> ----------------------------------------------------------
>
> Key: HDDS-16871
> URL: https://issues.apache.org/jira/browse/HDDS-16871
> Project: Apache Ozone
> Issue Type: Improvement
> Components: EC Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h2. Problem
> {\{KeyInputStream}} creates one \{{ECBlockInputStreamProxy}} per block, and
> each
> keeps its own \{{failedLocations}}. Nothing learned about a failing datanode
> while
> reading one block is used for the next block of the same key. When several
> blocks of a key have an index on the same unreachable datanode, each block:
> * starts with the healthy reader, because at creation the location still looks
> available;
> * waits for the connection attempts and retries to that datanode to fail;
> * fails over to the reconstruction reader (see sub-task 3 for the cost of the
> failover itself);
> * logs its own "Failing over to reconstruction read" warning.
> h2. HDFS behavior
> {\{DFSInputStream}} keeps the dead nodes it found for the life of the stream
> and
> skips them when choosing datanodes for later blocks; optionally a client-wide
> {\{DeadNodeDetector}} shares them across streams. "Could not obtain block"
> warnings are logged once per node (\{{warnedNodes}}).
> h2. Proposal
> * Keep a failed-datanode set in \{{KeyInputStream}} for the life of the key
> read
> and pass it to each EC block stream through the block stream factory.
> * \{{ECBlockInputStreamProxy.setReaderType}} counts locations on datanodes in
> that
> set as unavailable, so a later block starts in reconstruction mode (or the
> reconstruction reader does not select those indexes) without first trying the
> datanode.
> * Entries expire, so a datanode that comes back is used again; model the
> expiry
> on the write side's \{{ozone.client.exclude.nodes.expiry.time}}.
> * Only connection-level failures mark a datanode. A checksum error is a
> problem
> of one replica, not of the node, and stays per block.
> * Log the failover warning once per datanode per key read; keep later ones at
> debug.
> A client-wide dead-node detector, shared across streams, is out of scope.
> h2. Compatibility
> Client only. No wire or data change. A new expiry setting, if added, gets a
> default.
> h2. Testing
> * Unit, with a key of several blocks sharing a failing datanode: the first
> block
> tries the datanode and fails over; later blocks never call it and start in
> reconstruction mode.
> * After the expiry, the datanode is tried again.
> * A checksum failure on one block does not mark the datanode for other blocks.
> * The warning is logged once.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
