Andrey Yarovoy created HDDS-16867:
-------------------------------------
Summary: EC: ECBlockInputStream should read data cells from
datanodes in parallel
Key: HDDS-16867
URL: https://issues.apache.org/jira/browse/HDDS-16867
Project: Apache Ozone
Issue Type: Improvement
Components: EC Client
Reporter: Andrey Yarovoy
h2. Problem
On the healthy EC read path, \{{ECBlockInputStream}} reads one cell at a time,
serially. Each iteration of \{{readWithStrategy}} picks the data index for the
current position, opens its internal STANDALONE block stream, does one
synchronous read bounded by the cell, and only then moves to the next index:
{code:java}
// ECBlockInputStream.readWithStrategy
while (strategy.getTargetLength() > 0 && remaining() > 0) {
int currentIndex = currentStreamIndex();
BlockExtendedInputStream stream = getOrOpenStream(currentIndex);
int read = readFromStream(stream, strategy); // one datanode round trip
totalRead += read;
position += read;
}
{code}
A read that spans \{{N}} cells therefore costs \{{N}} serial datanode round
trips, although the cells are on \{{N}} different datanodes and do not depend on
each other. The reconstruction path is already parallel (HDDS-5952,
{\{ECBlockReconstructedStripeInputStream.loadDataBuffersFromStream}}); the
healthy path, which serves most reads, is not.
Through the filesystem, multi-cell reads come from callers that pass large
buffers. Note that for encrypted (TDE) keys only \{{read(ByteBuffer)}} passes
the
caller's buffer through: Hadoop's \{{CryptoInputStream.read(byte[], int, int)}}
reads through its internal buffer (\{{hadoop.security.crypto.buffer.size}},
8 KB by default), so byte-array reads on TDE keys never span more than one cell.
h2. HDFS behavior
{\{StripeReader.readStripe}} submits one read per needed data chunk of the
stripe to an \{{ExecutorCompletionService}} on the client-wide striped-reads
pool
(\{{dfs.client.read.striped.threadpool.size}}) and waits for them together.
h2. Proposal
When a \{{read}} call spans more than one cell, read the cells of different data
indexes concurrently and deliver the bytes in order.
* Split the target range into per-index segments. Segments on the same index
(from consecutive stripes) stay sequential on that index's stream, so the
fan-out is at most the number of data units.
* Keep a single-cell read on the existing synchronous path, with no executor
hand-off.
* Open the needed per-index streams under the stream monitor before fanning out
(\{{getOrOpenStream}} mutates \{{blockStreams[]}} and is not thread-safe). Each
task reads its segments into its own \{{duplicate()}} slice of the caller
buffer; the call joins all tasks before returning, so
{\{ECBlockInputStream.read}} keeps its \{{synchronized}} contract.
* On return, advance the position by the contiguous prefix that was read
successfully. A short read or failure on one index must not be hidden by
success on a later index.
* Keep the spare-location retry (\{{shouldRetryFailedRead}}) per index. When
spares cannot cover a failure, throw \{{BadDataLocationException}} carrying
every failed index of the call, so the proxy fails over once.
* Executor: either reuse the client's EC reconstruct executor
(\{{ozone.client.ec.reconstruct.stripe.read.pool.limit}}) or add a separate
bounded pool for healthy reads; to be decided in review. Either way, run the
task in the caller thread when the pool is saturated, as both the
reconstruction pool and HDFS do today.
* Configuration flag to turn the parallel path off.
Out of scope: reading ahead to the end of the stripe into an internal buffer
(HDFS \{{curStripeBuf}}). It changes how much is read for small reads and is a
separate decision.
h2. Compatibility
Client only. No wire or data change. A new configuration flag turns the
parallel path off; its default is to be decided in review.
h2. Testing
In \{{TestECBlockInputStream}} with \{{ECStreamTestUtil}}:
* a multi-cell read has reads in flight on several indexes at once (latch-based
test streams that block until all expected indexes have started);
* a single-cell read never uses the executor;
* the bytes returned match the written data for in-cell, cross-cell,
cross-stripe and partial-last-stripe ranges, for \{{rs-3-2}}, \{{rs-6-3}} and
{\{rs-10-4}};
* an error on one index: position advances by the contiguous prefix only, the
spare-retry path still works, and the exception reports all failed indexes;
* EOF and short reads behave as in the serial implementation;
* with the flag off, behavior is identical to today.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
