This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch orc
in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git
The following commit(s) were added to refs/heads/orc by this push:
new 00fa6d69e8e [opt](orc) Share parsed file tails across range readers
(#436)
00fa6d69e8e is described below
commit 00fa6d69e8ee1bfca05aa98037634bc42720d0db
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Nov 18 16:43:27 2026 +0800
[opt](orc) Share parsed file tails across range readers (#436)
* [opt](orc) Share parsed file tails across range readers
### Background
ORC range readers need file-wide footer metadata and absolute stripe row
prefixes. Physical split scheduling creates independent readers for one
immutable file.
### What problem does this PR solve?
Issue Number: None
Related PR: apache/doris#70664 (introduces physical child readers)
PR Type: Performance
Problem Summary:
Each physical child currently reparses the serialized file tail and
rebuilds all stripe row prefixes. Reuse a parsed immutable tail and initialize
each RowReader from its byte interval, keeping streams, schema, predicates and
metrics private.
### Details
- **Change map:** 225 production lines implement parsed-tail sharing,
bounded range/seek lookup, operation counters and corrupt-directory rejection;
142 test lines cover concurrency, lifetime and edge cases.
- **Why it was slow:** createReader reparses the FileTail protobuf and
RowReaderImpl scans every stripe for each child.
- **Optimization:** Share immutable Footer/PostScript and a
memory-pool-backed absolute row-prefix buffer; binary-search byte and row
boundaries. Reject an unordered stripe directory before binary search can
misselect data.
- **Results:** Static operation counts for an eight-stripe source plus
eight individual children: tail parses 9 to 1, prefix entries built 72 to 8,
RowReader range-accumulation visits 72 to 16. Binary-search comparisons are
additional logarithmic work. These counts are source-derived; no elapsed-time
or executed-test result is claimed.
- **Applicability and cost:** Helps sibling readers of one immutable file,
particularly uncoalesced stripes. Independent standalone readers retain their
existing API. A parsed tail retains one prefix buffer in its source memory
pool, which must outlive every shared handle. The new Reader API requires
consumers to rebuild. Metadata-only opens now build the prefix when the tail is
parsed; that path is not benchmarked.
### Testing
clang-format 16.0.6 and git diff --check passed. Added concurrent
range/seek, parent-destruction, serialized-tail compatibility, empty-file and
malformed-directory SDK tests. Build and runtime tests have not run because the
current worktree's installed dependencies point into a repository excluded by
AGENTS.md; permission to read them is pending.
### Release note
Reduce repeated ORC file-metadata initialization for physical split readers.
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Partially verified: compilation, runtime tests and
elapsed-time performance remain unverified
### AI Disclosure
- Tool: Codex
- Generated: Code, tests and this description
- Human review: Pending author review
* [opt](orc) Build shared stripe prefixes only when reading rows
### Background
Physical ORC children reuse parsed file tails and absolute stripe row
prefixes. Readers that only inspect metadata do not need row prefixes.
### What problem does this PR solve?
Issue Number: None
Related PR: apache/doris#70664 (review follow-up)
PR Type: Performance
Problem Summary:
Keep the shared prefix lazy so metadata-only opens avoid stripe traversal
and allocation. Concurrent children build one immutable prefix with
std::call_once, and an initialization failure remains retryable.
### Details
- **Why it was slow:** Eager prefix construction adds file-wide work to
metadata-only opens. Before parsed-tail sharing, each physical child also
reparsed the tail and rebuilt every prefix.
- **Optimization:** Build the source-pool-backed prefix only when a
RowReader needs it and publish only after success. Keep streams, mutable
schemas, predicates and metrics private.
- **Results:** The eight-stripe real TableReader fixture passes at one tail
parse, eight prefix entries and sixteen range visits. Temporarily restoring
repeated setup makes the same test fail at nine, seventy-two and seventy-two,
including the nonpositive-target case. SDK metadata-only opens build zero
prefix entries; cold concurrent children build eight in total and warm children
build none. These are measured operation counts, not elapsed-time claims.
- **Applicability and cost:** Reuses work within one immutable file
context. Byte/row range lookup uses logarithmic comparisons in addition to the
counted accumulation visits. The source memory pool must outlive its shared
handles. Serialized-tail callers remain supported; no session, wire or storage
format change is introduced by this follow-up.
### Testing
Local BE ASAN build through build.sh passed. run-be-ut.sh ran 255 related
tests with zero failures or skips after restoring the candidate from the
counterfactual. Coverage includes reader values/absolute row positions, read
intent, physical splitting, shared statistics, cold concurrent initialization,
warm reuse, parent destruction, seeks, empty files/ranges and repeated
malformed-directory errors. Changed-line BE clang-tidy, clang-format 16,
build-header hygiene and diff checks passe [...]
### Release note
Reduce repeated ORC physical-child metadata setup while preserving lazy
metadata-only opens.
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready
### AI Disclosure
- Tool: Codex
- Generated: Code, tests and this description
- Human review: Pending author review
---------
Co-authored-by: morningman <[email protected]>
---
c++/include/orc/Reader.hh | 16 +++++
c++/src/Options.hh | 10 +++
c++/src/Reader.cc | 178 +++++++++++++++++++++++++++++-----------------
c++/src/Reader.hh | 35 +++++++--
c++/test/TestReader.cc | 149 ++++++++++++++++++++++++++++++++++++++
5 files changed, 318 insertions(+), 70 deletions(-)
diff --git a/c++/include/orc/Reader.hh b/c++/include/orc/Reader.hh
index e45f1ce450e..0c44ae67faa 100644
--- a/c++/include/orc/Reader.hh
+++ b/c++/include/orc/Reader.hh
@@ -42,11 +42,17 @@ namespace orc {
struct RowReaderOptionsPrivate;
class InputStream;
class SharedStripeStatistics;
+ class SharedFileTail;
/**
* Expose the reader metrics including the latency and
* number of calls of the decompression/decoding/IO modules.
*/
struct ReaderMetrics {
+ // Successful file-tail parses, prefix entries built, and entries visited
to initialize
+ // RowReader ranges. A shared tail builds its prefix at most once, when
rows are needed.
+ std::atomic<uint64_t> FileTailParseCount{0};
+ std::atomic<uint64_t> StripeDirectoryEntryCount{0};
+ std::atomic<uint64_t> RowReaderStripeVisitCount{0};
// Metadata protobuf parse attempts and getNeedReadStripes evaluations
only.
// RowReader's range-local stripe/row-group PPD retains its existing
metrics.
std::atomic<uint64_t> MetadataParseCount{0};
@@ -109,6 +115,14 @@ namespace orc {
ReaderOptions&
setSharedStripeStatistics(std::shared_ptr<SharedStripeStatistics> statistics);
std::shared_ptr<SharedStripeStatistics> getSharedStripeStatistics() const;
+ /**
+ * Reuse the parsed tail and absolute stripe row prefixes of the same
immutable file.
+ * This takes precedence over the serialized tail. Streams, metrics and
mutable schemas
+ * remain reader-local. The source reader's memory pool must outlive the
shared handle.
+ */
+ ReaderOptions& setSharedFileTail(std::shared_ptr<const SharedFileTail>
tail);
+ std::shared_ptr<const SharedFileTail> getSharedFileTail() const;
+
/**
* Set the memory allocator.
*/
@@ -590,6 +604,8 @@ namespace orc {
*/
virtual std::string getSerializedFileTail() const = 0;
+ virtual std::shared_ptr<const SharedFileTail> getSharedFileTail() const =
0;
+
/**
* Get the type of the rows in the file. The top level is typically a
* struct.
diff --git a/c++/src/Options.hh b/c++/src/Options.hh
index 70819dbd3ce..0b90062749d 100644
--- a/c++/src/Options.hh
+++ b/c++/src/Options.hh
@@ -51,6 +51,7 @@ namespace orc {
std::string serializedTail;
ReaderMetrics* metrics;
std::shared_ptr<SharedStripeStatistics> stripeStatistics;
+ std::shared_ptr<const SharedFileTail> fileTail;
ReaderOptionsPrivate() {
tailLocation = std::numeric_limits<uint64_t>::max();
@@ -95,6 +96,15 @@ namespace orc {
return privateBits->stripeStatistics;
}
+ ReaderOptions& ReaderOptions::setSharedFileTail(std::shared_ptr<const
SharedFileTail> tail) {
+ privateBits->fileTail = std::move(tail);
+ return *this;
+ }
+
+ std::shared_ptr<const SharedFileTail> ReaderOptions::getSharedFileTail()
const {
+ return privateBits->fileTail;
+ }
+
ReaderOptions& ReaderOptions::setMemoryPool(MemoryPool& pool) {
privateBits->memoryPool = &pool;
return *this;
diff --git a/c++/src/Reader.cc b/c++/src/Reader.cc
index 1f7cb3716c2..dc888a2d0da 100644
--- a/c++/src/Reader.cc
+++ b/c++/src/Reader.cc
@@ -261,6 +261,29 @@ namespace orc {
buildTypeNameIdMap(contents->schema.get());
}
+ const DataBuffer<uint64_t>&
SharedFileTail::getFirstRowOfStripe(ReaderMetrics* metrics) const {
+ std::call_once(prefixOnce, [this, metrics] {
+ const auto stripeCount = static_cast<size_t>(footer->stripes_size());
+ auto prefixes = std::make_unique<DataBuffer<uint64_t>>(pool,
stripeCount);
+ uint64_t rowTotal = 0;
+ for (size_t stripe = 0; stripe < stripeCount; ++stripe) {
+ (*prefixes)[stripe] = rowTotal;
+ const auto& info = footer->stripes(static_cast<int>(stripe));
+ // Range lookup uses ordered offsets. Reject a corrupt directory such
as [100, 50]
+ // before a child could silently select the wrong stripes with binary
search.
+ if (stripe > 0 && info.offset() <
footer->stripes(static_cast<int>(stripe - 1)).offset()) {
+ throw ParseError("ORC stripe offsets are not ordered");
+ }
+ rowTotal += info.numberofrows();
+ }
+ firstRowOfStripe = std::move(prefixes);
+ if (metrics != nullptr) {
+ metrics->StripeDirectoryEntryCount.fetch_add(stripeCount,
std::memory_order_relaxed);
+ }
+ });
+ return *firstRowOfStripe;
+ }
+
RowReaderImpl::RowReaderImpl(std::shared_ptr<FileContents> _contents,
const RowReaderOptions& opts, const ORCFilter*
_filter,
const StringDictFilter* _stringDictFilter)
@@ -269,7 +292,7 @@ namespace orc {
throwOnHive11DecimalOverflow(opts.getThrowOnHive11DecimalOverflow()),
forcedScaleOnHive11Decimal(opts.getForcedScaleOnHive11Decimal()),
footer(contents->footer.get()),
- firstRowOfStripe(*contents->pool, 0),
+
firstRowOfStripe(contents->fileTail->getFirstRowOfStripe(contents->readerMetrics)),
enableEncodedBlock(opts.getEnableLazyDecoding()),
readerTimezone(getTimezoneByName(opts.getTimezoneName())),
filter(_filter),
@@ -282,28 +305,31 @@ namespace orc {
rowsInCurrentStripe = 0;
numRowGroupsInStripeRange = 0;
useTightNumericVector = opts.getUseTightNumericVector();
- uint64_t rowTotal = 0;
rowTotalInRange = 0;
- firstRowOfStripe.resize(numberOfStripes);
- for (size_t i = 0; i < numberOfStripes; ++i) {
- firstRowOfStripe[i] = rowTotal;
- proto::StripeInformation stripeInfo =
footer->stripes(static_cast<int>(i));
- rowTotal += stripeInfo.numberofrows();
- bool isStripeInRange = stripeInfo.offset() >= opts.getOffset() &&
- stripeInfo.offset() < opts.getOffset() +
opts.getLength();
- if (isStripeInRange) {
- rowTotalInRange += stripeInfo.numberofrows();
- if (i < currentStripe) {
- currentStripe = i;
- }
- if (i >= lastStripe) {
- lastStripe = i + 1;
- }
- if (footer->rowindexstride() > 0) {
- numRowGroupsInStripeRange +=
- (stripeInfo.numberofrows() + footer->rowindexstride() - 1) /
footer->rowindexstride();
- }
+ // Stripe offsets are ordered. For example, a one-stripe child of an
eight-stripe
+ // source reuses the eight absolute prefixes and visits only its own
directory entry.
+ const auto& stripes = footer->stripes();
+ const auto offsetLess = [](const proto::StripeInformation& stripe,
uint64_t offset) {
+ return stripe.offset() < offset;
+ };
+ const auto begin =
+ std::lower_bound(stripes.begin(), stripes.end(), opts.getOffset(),
offsetLess);
+ const auto end =
+ std::lower_bound(begin, stripes.end(), opts.getOffset() +
opts.getLength(), offsetLess);
+ if (begin != end) {
+ currentStripe = static_cast<uint64_t>(begin - stripes.begin());
+ lastStripe = static_cast<uint64_t>(end - stripes.begin());
+ }
+ if (contents->readerMetrics != nullptr) {
+ contents->readerMetrics->RowReaderStripeVisitCount.fetch_add(
+ static_cast<uint64_t>(end - begin), std::memory_order_relaxed);
+ }
+ for (auto stripe = begin; stripe != end; ++stripe) {
+ rowTotalInRange += stripe->numberofrows();
+ if (footer->rowindexstride() > 0) {
+ numRowGroupsInStripeRange +=
+ (stripe->numberofrows() + footer->rowindexstride() - 1) /
footer->rowindexstride();
}
}
firstStripe = currentStripe;
@@ -556,10 +582,11 @@ namespace orc {
return;
}
- uint64_t seekToStripe = 0;
- while (seekToStripe + 1 < lastStripe && firstRowOfStripe[seekToStripe + 1]
<= rowNumber) {
- seekToStripe++;
- }
+ // Absolute prefixes are shared with all physical children. A seek in a
late child
+ // must not walk every preceding sibling's prefix again.
+ uint64_t seekToStripe = static_cast<uint64_t>(
+ std::upper_bound(firstRowOfStripe.data(), firstRowOfStripe.data() +
lastStripe, rowNumber) -
+ firstRowOfStripe.data() - 1);
// seeking before the first stripe
if (seekToStripe < firstStripe) {
@@ -1783,52 +1810,73 @@ namespace orc {
contents->pool = options.getMemoryPool();
contents->errorStream = options.getErrorStream();
contents->readerMetrics = options.getReaderMetrics();
- std::string serializedFooter = options.getSerializedFileTail();
uint64_t fileLength;
uint64_t postscriptLength;
- if (serializedFooter.length() != 0) {
- // Parse the file tail from the serialized one.
- proto::FileTail tail;
- if (!tail.ParseFromString(serializedFooter)) {
- throw ParseError("Failed to parse the file tail from string");
- }
- contents->postscript =
std::make_unique<proto::PostScript>(tail.postscript());
- contents->footer = std::make_unique<proto::Footer>(tail.footer());
- fileLength = tail.filelength();
- postscriptLength = tail.postscriptlength();
+ contents->fileTail = options.getSharedFileTail();
+ // Physical siblings already have a parsed tail. Re-parsing its
serialization would
+ // repeat file-wide protobuf allocation and prefix construction for every
child.
+ if (contents->fileTail) {
+ contents->postscript = contents->fileTail->postscript;
+ contents->footer = contents->fileTail->footer;
+ fileLength = contents->fileTail->fileLength;
+ postscriptLength = contents->fileTail->postscriptLength;
} else {
- // figure out the size of the file using the option or filesystem
- fileLength = std::min(options.getTailLocation(),
static_cast<uint64_t>(stream->getLength()));
+ std::string serializedFooter = options.getSerializedFileTail();
+ if (serializedFooter.length() != 0) {
+ // Parse the file tail from the serialized one.
+ proto::FileTail tail;
+ if (!tail.ParseFromString(serializedFooter)) {
+ throw ParseError("Failed to parse the file tail from string");
+ }
+ contents->postscript =
std::make_unique<proto::PostScript>(tail.postscript());
+ contents->footer = std::make_unique<proto::Footer>(tail.footer());
+ fileLength = tail.filelength();
+ postscriptLength = tail.postscriptlength();
+ } else {
+ // figure out the size of the file using the option or filesystem
+ fileLength =
+ std::min(options.getTailLocation(),
static_cast<uint64_t>(stream->getLength()));
+
+ // read last bytes into buffer to get PostScript
+ uint64_t readSize = std::min(fileLength, DIRECTORY_SIZE_GUESS);
+ if (readSize < 4) {
+ throw ParseError("File size too small");
+ }
+ auto buffer = std::make_unique<DataBuffer<char>>(*contents->pool,
readSize);
+ stream->read(buffer->data(), readSize, fileLength - readSize);
+
+ postscriptLength = buffer->data()[readSize - 1] & 0xff;
+ contents->postscript = readPostscript(stream.get(), buffer.get(),
postscriptLength);
+ uint64_t footerSize = contents->postscript->footerlength();
+ uint64_t tailSize = 1 + postscriptLength + footerSize;
+ if (tailSize >= fileLength) {
+ std::stringstream msg;
+ msg << "Invalid ORC tailSize=" << tailSize << ", fileLength=" <<
fileLength;
+ throw ParseError(msg.str());
+ }
+ uint64_t footerOffset;
- // read last bytes into buffer to get PostScript
- uint64_t readSize = std::min(fileLength, DIRECTORY_SIZE_GUESS);
- if (readSize < 4) {
- throw ParseError("File size too small");
- }
- auto buffer = std::make_unique<DataBuffer<char>>(*contents->pool,
readSize);
- stream->read(buffer->data(), readSize, fileLength - readSize);
-
- postscriptLength = buffer->data()[readSize - 1] & 0xff;
- contents->postscript = readPostscript(stream.get(), buffer.get(),
postscriptLength);
- uint64_t footerSize = contents->postscript->footerlength();
- uint64_t tailSize = 1 + postscriptLength + footerSize;
- if (tailSize >= fileLength) {
- std::stringstream msg;
- msg << "Invalid ORC tailSize=" << tailSize << ", fileLength=" <<
fileLength;
- throw ParseError(msg.str());
- }
- uint64_t footerOffset;
+ if (tailSize > readSize) {
+ buffer->resize(footerSize);
+ stream->read(buffer->data(), footerSize, fileLength - tailSize);
+ footerOffset = 0;
+ } else {
+ footerOffset = readSize - tailSize;
+ }
- if (tailSize > readSize) {
- buffer->resize(footerSize);
- stream->read(buffer->data(), footerSize, fileLength - tailSize);
- footerOffset = 0;
- } else {
- footerOffset = readSize - tailSize;
+ contents->footer =
+ readFooter(stream.get(), buffer.get(), footerOffset,
*contents->postscript,
+ *contents->pool, contents->readerMetrics);
}
-
- contents->footer = readFooter(stream.get(), buffer.get(), footerOffset,
*contents->postscript,
- *contents->pool, contents->readerMetrics);
+ if (contents->readerMetrics != nullptr) {
+ contents->readerMetrics->FileTailParseCount.fetch_add(1,
std::memory_order_relaxed);
+ }
+ auto tail = std::make_shared<SharedFileTail>(*contents->pool);
+ tail->postscript = contents->postscript;
+ tail->footer = contents->footer;
+ tail->fileLength = fileLength;
+ tail->postscriptLength = postscriptLength;
+ contents->fileTail = std::move(tail);
}
contents->isDecimalAsLong = false;
if (contents->postscript->version_size() == 2) {
diff --git a/c++/src/Reader.hh b/c++/src/Reader.hh
index 537a3a5908a..94b1e5b31ba 100644
--- a/c++/src/Reader.hh
+++ b/c++/src/Reader.hh
@@ -95,11 +95,32 @@ namespace orc {
std::shared_ptr<const proto::Metadata> metadata;
};
+ // The parsed tail is immutable after publication. Prefixes are built once
on first RowReader
+ // use, so metadata-only readers do not allocate them. The source allocator
must outlive this
+ // handle; no reader, stream, mutable schema or metrics are retained.
+ class SharedFileTail {
+ public:
+ explicit SharedFileTail(MemoryPool& memoryPool) : pool(memoryPool) {}
+
+ const DataBuffer<uint64_t>& getFirstRowOfStripe(ReaderMetrics* metrics)
const;
+
+ std::shared_ptr<const proto::PostScript> postscript;
+ std::shared_ptr<const proto::Footer> footer;
+ uint64_t fileLength;
+ uint64_t postscriptLength;
+
+ private:
+ MemoryPool& pool;
+ mutable std::once_flag prefixOnce;
+ mutable std::unique_ptr<DataBuffer<uint64_t>> firstRowOfStripe;
+ };
+
/** State shared between one Reader and its RowReaders. */
struct FileContents {
std::unique_ptr<InputStream> stream;
- std::unique_ptr<proto::PostScript> postscript;
- std::unique_ptr<proto::Footer> footer;
+ std::shared_ptr<const proto::PostScript> postscript;
+ std::shared_ptr<const proto::Footer> footer;
+ std::shared_ptr<const SharedFileTail> fileTail;
std::unique_ptr<Type> schema;
uint64_t blockSize;
CompressionKind compression;
@@ -181,8 +202,8 @@ namespace orc {
std::vector<bool> selectedColumns;
// footer
- proto::Footer* footer;
- DataBuffer<uint64_t> firstRowOfStripe;
+ const proto::Footer* footer;
+ const DataBuffer<uint64_t>& firstRowOfStripe;
mutable std::unique_ptr<Type> selectedSchema;
bool skipBloomFilters;
@@ -325,7 +346,7 @@ namespace orc {
const uint64_t postscriptLength;
// footer
- proto::Footer* footer;
+ const proto::Footer* footer;
uint64_t numberOfStripes;
std::unique_ptr<SargsApplier> sargsApplier;
std::vector<int> getNeedReadStripes(const RowReaderOptions& opts) override;
@@ -411,6 +432,10 @@ namespace orc {
std::string getSerializedFileTail() const override;
+ std::shared_ptr<const SharedFileTail> getSharedFileTail() const override {
+ return contents->fileTail;
+ }
+
const Type& getType() const override;
bool hasCorrectStatistics() const override;
diff --git a/c++/test/TestReader.cc b/c++/test/TestReader.cc
index f709f693f10..33a307035cd 100644
--- a/c++/test/TestReader.cc
+++ b/c++/test/TestReader.cc
@@ -17,6 +17,7 @@
*/
#include <cstring>
+#include <thread>
#include "Reader.hh"
#include "orc/Reader.hh"
@@ -34,6 +35,154 @@ namespace orc {
static const int DEFAULT_MEM_STREAM_SIZE = 1024 * 1024; // 1M
+ TEST(TestReader, SharedFileTailKeepsIndependentRangeReaders) {
+ constexpr uint64_t STRIPES = 8;
+ // Exceed the RLE encoder's buffer so every add() has encoded bytes to
trigger a stripe flush.
+ constexpr uint64_t ROWS = 3000;
+ MemoryOutputStream stream(DEFAULT_MEM_STREAM_SIZE);
+ auto type = Type::buildTypeFromString("struct<id:bigint>");
+ WriterOptions writeOptions;
+
writeOptions.setStripeSize(1).setCompression(CompressionKind_NONE).setRowIndexStride(0);
+ auto writer = createWriter(*type, &stream, writeOptions);
+ auto batch = writer->createRowBatch(ROWS);
+ auto& root = dynamic_cast<StructVectorBatch&>(*batch);
+ auto& ids = dynamic_cast<LongVectorBatch&>(*root.fields[0]);
+ for (uint64_t stripe = 0; stripe < STRIPES; ++stripe) {
+ for (uint64_t row = 0; row < ROWS; ++row) {
+ ids.data[row] = static_cast<int64_t>(stripe * ROWS + row);
+ }
+ root.numElements = ids.numElements = ROWS;
+ writer->add(*batch);
+ }
+ writer->close();
+
+ std::shared_ptr<const SharedFileTail> shared;
+ std::vector<uint64_t> offsets;
+ std::string serialized;
+ // No parent stream, metrics or schema survives this scope.
+ {
+ ReaderMetrics metrics;
+ ReaderOptions options;
+ options.setReaderMetrics(&metrics);
+ auto parent = createReader(
+ std::make_unique<MemoryInputStream>(stream.getData(),
stream.getLength()), options);
+ ASSERT_EQ(STRIPES, parent->getNumberOfStripes());
+ shared = parent->getSharedFileTail();
+ serialized = parent->getSerializedFileTail();
+ for (uint64_t stripe = 0; stripe < STRIPES; ++stripe) {
+ offsets.push_back(parent->getStripe(stripe)->getOffset());
+ }
+ EXPECT_EQ(1, metrics.FileTailParseCount.load());
+ EXPECT_EQ(0, metrics.StripeDirectoryEntryCount.load());
+ }
+ ASSERT_NE(nullptr, shared);
+ offsets.push_back(stream.getLength());
+
+ // Independent children race on immutable metadata, first individually and
then coalesced.
+ for (uint64_t width : {uint64_t{1}, uint64_t{4}}) {
+ std::atomic<uint64_t> ready{0};
+ std::atomic<bool> start{false};
+ std::atomic<uint64_t> prefixEntries{0};
+ std::vector<std::thread> children;
+ for (uint64_t first = 0; first < STRIPES; first += width) {
+ children.emplace_back([&, first] {
+ ready.fetch_add(1, std::memory_order_release);
+ while (!start.load(std::memory_order_acquire)) {
+ std::this_thread::yield();
+ }
+ ReaderMetrics metrics;
+ const ReaderOptions options = ReaderOptions()
+ .setSharedFileTail(shared)
+ .setReaderMetrics(&metrics)
+ .setSerializedFileTail("not a
serialized tail");
+ auto reader = createReader(
+ std::make_unique<MemoryInputStream>(stream.getData(),
stream.getLength()), options);
+ EXPECT_EQ(shared, reader->getSharedFileTail());
+ auto rows = reader->createRowReader(
+ RowReaderOptions().range(offsets[first], offsets[first + width]
- offsets[first]));
+ EXPECT_EQ(width * ROWS, rows->getNumberOfRows());
+ EXPECT_EQ(0, metrics.FileTailParseCount.load());
+ EXPECT_EQ(width, metrics.RowReaderStripeVisitCount.load());
+ auto data = rows->createRowBatch(37);
+ uint64_t expected = first * ROWS;
+ while (rows->next(*data)) {
+ EXPECT_EQ(expected, rows->getRowNumber());
+ auto& values =
+
dynamic_cast<LongVectorBatch&>(*dynamic_cast<StructVectorBatch&>(*data).fields[0]);
+ for (uint64_t row = 0; row < data->numElements; ++row) {
+ EXPECT_EQ(static_cast<int64_t>(expected++), values.data[row]);
+ }
+ }
+ EXPECT_EQ((first + width) * ROWS, expected);
+ const auto seekRow = (first + width) * ROWS - 1;
+ rows->seekToRow(seekRow);
+ ASSERT_TRUE(rows->next(*data));
+ EXPECT_EQ(seekRow, rows->getRowNumber());
+ EXPECT_EQ(1, data->numElements);
+ auto& values =
+
dynamic_cast<LongVectorBatch&>(*dynamic_cast<StructVectorBatch&>(*data).fields[0]);
+ EXPECT_EQ(static_cast<int64_t>(seekRow), values.data[0]);
+ prefixEntries.fetch_add(metrics.StripeDirectoryEntryCount.load(),
+ std::memory_order_relaxed);
+ });
+ }
+ while (ready.load(std::memory_order_acquire) != STRIPES / width) {
+ std::this_thread::yield();
+ }
+ start.store(true, std::memory_order_release);
+ for (auto& child : children) {
+ child.join();
+ }
+ EXPECT_EQ(width == 1 ? STRIPES : 0, prefixEntries.load());
+ }
+
+ // The existing serialized-tail API still builds an independent parsed
context.
+ ReaderMetrics metrics;
+ const ReaderOptions options =
+
ReaderOptions().setSerializedFileTail(serialized).setReaderMetrics(&metrics);
+ auto reader = createReader(
+ std::make_unique<MemoryInputStream>(stream.getData(),
stream.getLength()), options);
+ EXPECT_NE(shared, reader->getSharedFileTail());
+ EXPECT_EQ(1, metrics.FileTailParseCount.load());
+ EXPECT_EQ(0, metrics.StripeDirectoryEntryCount.load());
+ EXPECT_EQ(STRIPES * ROWS, reader->getNumberOfRows());
+ EXPECT_EQ(0, metrics.StripeDirectoryEntryCount.load());
+ auto empty = reader->createRowReader(RowReaderOptions().range(offsets[2],
0));
+ EXPECT_EQ(STRIPES, metrics.StripeDirectoryEntryCount.load());
+ EXPECT_EQ(0, empty->getNumberOfRows());
+ EXPECT_EQ(0, metrics.RowReaderStripeVisitCount.load());
+ EXPECT_FALSE(empty->next(*empty->createRowBatch(1)));
+
+ proto::FileTail corrupt;
+ ASSERT_TRUE(corrupt.ParseFromString(serialized));
+ corrupt.mutable_footer()->mutable_stripes(1)->set_offset(offsets[0] - 1);
+ const ReaderOptions corruptOptions =
+ ReaderOptions().setSerializedFileTail(corrupt.SerializeAsString());
+ auto corruptReader = createReader(
+ std::make_unique<MemoryInputStream>(stream.getData(),
stream.getLength()), corruptOptions);
+ EXPECT_THROW(corruptReader->createRowReader(), ParseError);
+ EXPECT_THROW(corruptReader->createRowReader(), ParseError);
+ }
+
+ TEST(TestReader, SharedFileTailSupportsEmptyFiles) {
+ MemoryOutputStream stream(DEFAULT_MEM_STREAM_SIZE);
+ auto type = Type::buildTypeFromString("struct<id:bigint>");
+ auto writer = createWriter(*type, &stream, WriterOptions());
+ writer->close();
+ auto parent = createReader(
+ std::make_unique<MemoryInputStream>(stream.getData(),
stream.getLength()), ReaderOptions());
+ const ReaderOptions options =
ReaderOptions().setSharedFileTail(parent->getSharedFileTail());
+ parent.reset();
+ auto reader = createReader(
+ std::make_unique<MemoryInputStream>(stream.getData(),
stream.getLength()), options);
+ auto rows = reader->createRowReader();
+ EXPECT_EQ(0, rows->getNumberOfRows());
+ auto batch = rows->createRowBatch(1);
+ EXPECT_FALSE(rows->next(*batch));
+ rows->seekToRow(0);
+ EXPECT_FALSE(rows->next(*batch));
+ }
+
TEST(TestReader, testWriterVersions) {
EXPECT_EQ("original", writerVersionToString(WriterVersion_ORIGINAL));
EXPECT_EQ("HIVE-8732", writerVersionToString(WriterVersion_HIVE_8732));
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]