Andrey Yarovoy created HDDS-16866:
-------------------------------------
Summary: EC: Seek on a reconstruction-mode EC block reads a full
stripe eagerly
Key: HDDS-16866
URL: https://issues.apache.org/jira/browse/HDDS-16866
Project: Apache Ozone
Issue Type: Bug
Components: EC Client
Reporter: Andrey Yarovoy
h2. Problem
{\{ECBlockReconstructedInputStream.seek(pos)}} reads data from the datanodes
before any read is requested. After the EOF check it seeks the stripe reader to
the start of the target stripe and calls \{{readAndSeekStripe}}, which allocates
the stripe buffers and reads (and decodes) the whole stripe:
{code:java}
// ECBlockReconstructedInputStream.seek
stripeReader.seek(stripeNum * stripeSize);
readAndSeekStripe(partial); // allocateBuffers() + readStripe()
position = pos;
{code}
So every seek on a block in reconstruction mode costs one full stripe of
datanode reads plus a decode, whether or not the caller then reads, and a
second seek into the same stripe reads it again.
Two callers turn this into repeated reads of data nobody asked for:
* \{{MultipartInputStream.seek}} calls \{{seek(0)}} on the previously current
part
when the part index changes, and on *every* part after the target part. On an
EC key, each such part whose block is in reconstruction mode reads its first
stripe, on every key-level seek.
* \{{OzoneFSInputStream}} serves positional reads on EC keys through the
synchronized seek - read - restore fallback (EC keys are not admitted to
stateless pread), so each pread performs at least two key-level seeks.
A block is in reconstruction mode when \{{ECBlockInputStreamProxy}} sees fewer
available data locations than it needs when the block stream is created (for
example a datanode in the pipeline is dead, stale or being decommissioned), or
after it fails over from \{{ECBlockInputStream}} on a read error.
h2. HDFS behavior
{\{DFSStripedInputStream.seek}} is lazy. If the target lies inside the stripe
already held in \{{curStripeBuf}} it only repositions the buffer; otherwise it
records the target (\{{pos = targetPos; blockEnd = -1}}) and the next read loads
the stripe. A seek by itself performs no datanode I/O.
h2. Reproduction
Unit test with \{{ECStreamTestUtil.TestBlockInputStreamFactory}} wrapped to
count
bytes read from the per-index streams, \{{rs-3-2-1024k}}, one full block group,
data index 3 has no location (so the proxy selects the reconstruction reader):
||Action||Bytes read from datanode streams||
|\{{seek(0)}} on a fresh \{{ECBlockReconstructedInputStream}}|3 MiB (one
stripe)|
|second \{{seek(0)}}|3 MiB more|
|\{{seek}} to an unaligned offset in stripe 2|3 MiB more|
|4 KB \{{read}} after that seek|0 more|
|\{{MultipartInputStream}} of [healthy part, reconstruction-mode part], 4 seeks
that stay inside the first part|12 MiB from the second part, which is never
read|
h2. Proposal
Make \{{ECBlockReconstructedInputStream.seek}} lazy:
* Keep the EOF / negative-position check in \{{seek}} so the exception is still
thrown at seek time.
* Record the new position and mark the stripe as not loaded; do not call
{\{stripeReader.seek}} or \{{readAndSeekStripe}}. The class already has this
pattern for \{{unbuffer()}}: it sets \{{unBuffered = true}} and \{{read}} does
the
real seek on the next call. Reuse it, with the read path calling the eager
stripe load directly (not the now-lazy public \{{seek}}).
* Optionally, when the target is inside the stripe already loaded, only
reposition the stripe buffers (the HDFS in-stripe case).
No change to \{{MultipartInputStream}} is required: once a seek is free, its
{\{seek(0)}} calls on other parts cost nothing.
h2. Compatibility
Client only. No wire, configuration or data change.
h2. Testing
In \{{TestECBlockReconstructedInputStream}} with a counting stream factory:
* \{{seek}} alone, and repeated \{{seek(0)}}, read nothing from the datanode
streams;
* \{{seek}} + \{{read}} reads exactly one stripe and returns the right bytes,
for
in-cell, cross-cell and last-partial-stripe offsets;
* a seek inside the loaded stripe does not read again (if the optional part is
done);
* \{{seek}} past the end of the block still throws \{{EOFException}} from
{\{seek}};
* \{{seek}} after \{{unbuffer}} still works.
At the key level, a \{{MultipartInputStream}} with a reconstruction-mode part
after a healthy part: seeks that stay in the first part read nothing from the
second.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
