[
https://issues.apache.org/jira/browse/HDDS-16866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-16866:
-------------------------------------
Assignee: Andrey Yarovoy
> EC: Seek on a reconstruction-mode EC block reads a full stripe eagerly
> ----------------------------------------------------------------------
>
> Key: HDDS-16866
> URL: https://issues.apache.org/jira/browse/HDDS-16866
> Project: Apache Ozone
> Issue Type: Bug
> Components: EC Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h2. Problem
> {\{ECBlockReconstructedInputStream.seek(pos)}} reads data from the datanodes
> before any read is requested. After the EOF check it seeks the stripe reader
> to
> the start of the target stripe and calls \{{readAndSeekStripe}}, which
> allocates
> the stripe buffers and reads (and decodes) the whole stripe:
> {code:java}
> // ECBlockReconstructedInputStream.seek
> stripeReader.seek(stripeNum * stripeSize);
> readAndSeekStripe(partial); // allocateBuffers() + readStripe()
> position = pos;
> {code}
> So every seek on a block in reconstruction mode costs one full stripe of
> datanode reads plus a decode, whether or not the caller then reads, and a
> second seek into the same stripe reads it again.
> Two callers turn this into repeated reads of data nobody asked for:
> * \{{MultipartInputStream.seek}} calls \{{seek(0)}} on the previously current
> part
> when the part index changes, and on *every* part after the target part. On an
> EC key, each such part whose block is in reconstruction mode reads its first
> stripe, on every key-level seek.
> * \{{OzoneFSInputStream}} serves positional reads on EC keys through the
> synchronized seek - read - restore fallback (EC keys are not admitted to
> stateless pread), so each pread performs at least two key-level seeks.
> A block is in reconstruction mode when \{{ECBlockInputStreamProxy}} sees fewer
> available data locations than it needs when the block stream is created (for
> example a datanode in the pipeline is dead, stale or being decommissioned), or
> after it fails over from \{{ECBlockInputStream}} on a read error.
> h2. HDFS behavior
> {\{DFSStripedInputStream.seek}} is lazy. If the target lies inside the stripe
> already held in \{{curStripeBuf}} it only repositions the buffer; otherwise it
> records the target (\{{pos = targetPos; blockEnd = -1}}) and the next read
> loads
> the stripe. A seek by itself performs no datanode I/O.
> h2. Reproduction
> Unit test with \{{ECStreamTestUtil.TestBlockInputStreamFactory}} wrapped to
> count
> bytes read from the per-index streams, \{{rs-3-2-1024k}}, one full block
> group,
> data index 3 has no location (so the proxy selects the reconstruction reader):
> ||Action||Bytes read from datanode streams||
> |\{{seek(0)}} on a fresh \{{ECBlockReconstructedInputStream}}|3 MiB (one
> stripe)|
> |second \{{seek(0)}}|3 MiB more|
> |\{{seek}} to an unaligned offset in stripe 2|3 MiB more|
> |4 KB \{{read}} after that seek|0 more|
> |\{{MultipartInputStream}} of [healthy part, reconstruction-mode part], 4
> seeks that stay inside the first part|12 MiB from the second part, which is
> never read|
> h2. Proposal
> Make \{{ECBlockReconstructedInputStream.seek}} lazy:
> * Keep the EOF / negative-position check in \{{seek}} so the exception is
> still
> thrown at seek time.
> * Record the new position and mark the stripe as not loaded; do not call
> {\{stripeReader.seek}} or \{{readAndSeekStripe}}. The class already has this
> pattern for \{{unbuffer()}}: it sets \{{unBuffered = true}} and \{{read}}
> does the
> real seek on the next call. Reuse it, with the read path calling the eager
> stripe load directly (not the now-lazy public \{{seek}}).
> * Optionally, when the target is inside the stripe already loaded, only
> reposition the stripe buffers (the HDFS in-stripe case).
> No change to \{{MultipartInputStream}} is required: once a seek is free, its
> {\{seek(0)}} calls on other parts cost nothing.
> h2. Compatibility
> Client only. No wire, configuration or data change.
> h2. Testing
> In \{{TestECBlockReconstructedInputStream}} with a counting stream factory:
> * \{{seek}} alone, and repeated \{{seek(0)}}, read nothing from the datanode
> streams;
> * \{{seek}} + \{{read}} reads exactly one stripe and returns the right bytes,
> for
> in-cell, cross-cell and last-partial-stripe offsets;
> * a seek inside the loaded stripe does not read again (if the optional part is
> done);
> * \{{seek}} past the end of the block still throws \{{EOFException}} from
> {\{seek}};
> * \{{seek}} after \{{unbuffer}} still works.
> At the key level, a \{{MultipartInputStream}} with a reconstruction-mode part
> after a healthy part: seeks that stay in the first part read nothing from the
> second.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
