This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch orc
in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git


The following commit(s) were added to refs/heads/orc by this push:
     new 00fa6d69e8e [opt](orc) Share parsed file tails across range readers 
(#436)
00fa6d69e8e is described below

commit 00fa6d69e8ee1bfca05aa98037634bc42720d0db
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Nov 18 16:43:27 2026 +0800

    [opt](orc) Share parsed file tails across range readers (#436)
    
    * [opt](orc) Share parsed file tails across range readers
    
    ### Background
    
    ORC range readers need file-wide footer metadata and absolute stripe row 
prefixes. Physical split scheduling creates independent readers for one 
immutable file.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: apache/doris#70664 (introduces physical child readers)
    PR Type: Performance
    
    Problem Summary:
    
    Each physical child currently reparses the serialized file tail and 
rebuilds all stripe row prefixes. Reuse a parsed immutable tail and initialize 
each RowReader from its byte interval, keeping streams, schema, predicates and 
metrics private.
    
    ### Details
    
    - **Change map:** 225 production lines implement parsed-tail sharing, 
bounded range/seek lookup, operation counters and corrupt-directory rejection; 
142 test lines cover concurrency, lifetime and edge cases.
    - **Why it was slow:** createReader reparses the FileTail protobuf and 
RowReaderImpl scans every stripe for each child.
    - **Optimization:** Share immutable Footer/PostScript and a 
memory-pool-backed absolute row-prefix buffer; binary-search byte and row 
boundaries. Reject an unordered stripe directory before binary search can 
misselect data.
    - **Results:** Static operation counts for an eight-stripe source plus 
eight individual children: tail parses 9 to 1, prefix entries built 72 to 8, 
RowReader range-accumulation visits 72 to 16. Binary-search comparisons are 
additional logarithmic work. These counts are source-derived; no elapsed-time 
or executed-test result is claimed.
    - **Applicability and cost:** Helps sibling readers of one immutable file, 
particularly uncoalesced stripes. Independent standalone readers retain their 
existing API. A parsed tail retains one prefix buffer in its source memory 
pool, which must outlive every shared handle. The new Reader API requires 
consumers to rebuild. Metadata-only opens now build the prefix when the tail is 
parsed; that path is not benchmarked.
    
    ### Testing
    
    clang-format 16.0.6 and git diff --check passed. Added concurrent 
range/seek, parent-destruction, serialized-tail compatibility, empty-file and 
malformed-directory SDK tests. Build and runtime tests have not run because the 
current worktree's installed dependencies point into a repository excluded by 
AGENTS.md; permission to read them is pending.
    
    ### Release note
    
    Reduce repeated ORC file-metadata initialization for physical split readers.
    
    ### Check List (For Author)
    
    - Behavior changed: No
    - Does this need documentation: No
    - Pre-submission status: Partially verified: compilation, runtime tests and 
elapsed-time performance remain unverified
    
    ### AI Disclosure
    
    - Tool: Codex
    - Generated: Code, tests and this description
    - Human review: Pending author review
    
    * [opt](orc) Build shared stripe prefixes only when reading rows
    
    ### Background
    
    Physical ORC children reuse parsed file tails and absolute stripe row 
prefixes. Readers that only inspect metadata do not need row prefixes.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: apache/doris#70664 (review follow-up)
    PR Type: Performance
    
    Problem Summary:
    
    Keep the shared prefix lazy so metadata-only opens avoid stripe traversal 
and allocation. Concurrent children build one immutable prefix with 
std::call_once, and an initialization failure remains retryable.
    
    ### Details
    
    - **Why it was slow:** Eager prefix construction adds file-wide work to 
metadata-only opens. Before parsed-tail sharing, each physical child also 
reparsed the tail and rebuilt every prefix.
    - **Optimization:** Build the source-pool-backed prefix only when a 
RowReader needs it and publish only after success. Keep streams, mutable 
schemas, predicates and metrics private.
    - **Results:** The eight-stripe real TableReader fixture passes at one tail 
parse, eight prefix entries and sixteen range visits. Temporarily restoring 
repeated setup makes the same test fail at nine, seventy-two and seventy-two, 
including the nonpositive-target case. SDK metadata-only opens build zero 
prefix entries; cold concurrent children build eight in total and warm children 
build none. These are measured operation counts, not elapsed-time claims.
    - **Applicability and cost:** Reuses work within one immutable file 
context. Byte/row range lookup uses logarithmic comparisons in addition to the 
counted accumulation visits. The source memory pool must outlive its shared 
handles. Serialized-tail callers remain supported; no session, wire or storage 
format change is introduced by this follow-up.
    
    ### Testing
    
    Local BE ASAN build through build.sh passed. run-be-ut.sh ran 255 related 
tests with zero failures or skips after restoring the candidate from the 
counterfactual. Coverage includes reader values/absolute row positions, read 
intent, physical splitting, shared statistics, cold concurrent initialization, 
warm reuse, parent destruction, seeks, empty files/ranges and repeated 
malformed-directory errors. Changed-line BE clang-tidy, clang-format 16, 
build-header hygiene and diff checks passe [...]
    
    ### Release note
    
    Reduce repeated ORC physical-child metadata setup while preserving lazy 
metadata-only opens.
    
    ### Check List (For Author)
    
    - Behavior changed: No
    - Does this need documentation: No
    - Pre-submission status: Ready
    
    ### AI Disclosure
    
    - Tool: Codex
    - Generated: Code, tests and this description
    - Human review: Pending author review
    
    ---------
    
    Co-authored-by: morningman <[email protected]>
---
 c++/include/orc/Reader.hh |  16 +++++
 c++/src/Options.hh        |  10 +++
 c++/src/Reader.cc         | 178 +++++++++++++++++++++++++++++-----------------
 c++/src/Reader.hh         |  35 +++++++--
 c++/test/TestReader.cc    | 149 ++++++++++++++++++++++++++++++++++++++
 5 files changed, 318 insertions(+), 70 deletions(-)

diff --git a/c++/include/orc/Reader.hh b/c++/include/orc/Reader.hh
index e45f1ce450e..0c44ae67faa 100644
--- a/c++/include/orc/Reader.hh
+++ b/c++/include/orc/Reader.hh
@@ -42,11 +42,17 @@ namespace orc {
   struct RowReaderOptionsPrivate;
   class InputStream;
   class SharedStripeStatistics;
+  class SharedFileTail;
   /**
    * Expose the reader metrics including the latency and
    * number of calls of the decompression/decoding/IO modules.
    */
   struct ReaderMetrics {
+    // Successful file-tail parses, prefix entries built, and entries visited 
to initialize
+    // RowReader ranges. A shared tail builds its prefix at most once, when 
rows are needed.
+    std::atomic<uint64_t> FileTailParseCount{0};
+    std::atomic<uint64_t> StripeDirectoryEntryCount{0};
+    std::atomic<uint64_t> RowReaderStripeVisitCount{0};
     // Metadata protobuf parse attempts and getNeedReadStripes evaluations 
only.
     // RowReader's range-local stripe/row-group PPD retains its existing 
metrics.
     std::atomic<uint64_t> MetadataParseCount{0};
@@ -109,6 +115,14 @@ namespace orc {
     ReaderOptions& 
setSharedStripeStatistics(std::shared_ptr<SharedStripeStatistics> statistics);
     std::shared_ptr<SharedStripeStatistics> getSharedStripeStatistics() const;
 
+    /**
+     * Reuse the parsed tail and absolute stripe row prefixes of the same 
immutable file.
+     * This takes precedence over the serialized tail. Streams, metrics and 
mutable schemas
+     * remain reader-local. The source reader's memory pool must outlive the 
shared handle.
+     */
+    ReaderOptions& setSharedFileTail(std::shared_ptr<const SharedFileTail> 
tail);
+    std::shared_ptr<const SharedFileTail> getSharedFileTail() const;
+
     /**
      * Set the memory allocator.
      */
@@ -590,6 +604,8 @@ namespace orc {
      */
     virtual std::string getSerializedFileTail() const = 0;
 
+    virtual std::shared_ptr<const SharedFileTail> getSharedFileTail() const = 
0;
+
     /**
      * Get the type of the rows in the file. The top level is typically a
      * struct.
diff --git a/c++/src/Options.hh b/c++/src/Options.hh
index 70819dbd3ce..0b90062749d 100644
--- a/c++/src/Options.hh
+++ b/c++/src/Options.hh
@@ -51,6 +51,7 @@ namespace orc {
     std::string serializedTail;
     ReaderMetrics* metrics;
     std::shared_ptr<SharedStripeStatistics> stripeStatistics;
+    std::shared_ptr<const SharedFileTail> fileTail;
 
     ReaderOptionsPrivate() {
       tailLocation = std::numeric_limits<uint64_t>::max();
@@ -95,6 +96,15 @@ namespace orc {
     return privateBits->stripeStatistics;
   }
 
+  ReaderOptions& ReaderOptions::setSharedFileTail(std::shared_ptr<const 
SharedFileTail> tail) {
+    privateBits->fileTail = std::move(tail);
+    return *this;
+  }
+
+  std::shared_ptr<const SharedFileTail> ReaderOptions::getSharedFileTail() 
const {
+    return privateBits->fileTail;
+  }
+
   ReaderOptions& ReaderOptions::setMemoryPool(MemoryPool& pool) {
     privateBits->memoryPool = &pool;
     return *this;
diff --git a/c++/src/Reader.cc b/c++/src/Reader.cc
index 1f7cb3716c2..dc888a2d0da 100644
--- a/c++/src/Reader.cc
+++ b/c++/src/Reader.cc
@@ -261,6 +261,29 @@ namespace orc {
     buildTypeNameIdMap(contents->schema.get());
   }
 
+  const DataBuffer<uint64_t>& 
SharedFileTail::getFirstRowOfStripe(ReaderMetrics* metrics) const {
+    std::call_once(prefixOnce, [this, metrics] {
+      const auto stripeCount = static_cast<size_t>(footer->stripes_size());
+      auto prefixes = std::make_unique<DataBuffer<uint64_t>>(pool, 
stripeCount);
+      uint64_t rowTotal = 0;
+      for (size_t stripe = 0; stripe < stripeCount; ++stripe) {
+        (*prefixes)[stripe] = rowTotal;
+        const auto& info = footer->stripes(static_cast<int>(stripe));
+        // Range lookup uses ordered offsets. Reject a corrupt directory such 
as [100, 50]
+        // before a child could silently select the wrong stripes with binary 
search.
+        if (stripe > 0 && info.offset() < 
footer->stripes(static_cast<int>(stripe - 1)).offset()) {
+          throw ParseError("ORC stripe offsets are not ordered");
+        }
+        rowTotal += info.numberofrows();
+      }
+      firstRowOfStripe = std::move(prefixes);
+      if (metrics != nullptr) {
+        metrics->StripeDirectoryEntryCount.fetch_add(stripeCount, 
std::memory_order_relaxed);
+      }
+    });
+    return *firstRowOfStripe;
+  }
+
   RowReaderImpl::RowReaderImpl(std::shared_ptr<FileContents> _contents,
                                const RowReaderOptions& opts, const ORCFilter* 
_filter,
                                const StringDictFilter* _stringDictFilter)
@@ -269,7 +292,7 @@ namespace orc {
         throwOnHive11DecimalOverflow(opts.getThrowOnHive11DecimalOverflow()),
         forcedScaleOnHive11Decimal(opts.getForcedScaleOnHive11Decimal()),
         footer(contents->footer.get()),
-        firstRowOfStripe(*contents->pool, 0),
+        
firstRowOfStripe(contents->fileTail->getFirstRowOfStripe(contents->readerMetrics)),
         enableEncodedBlock(opts.getEnableLazyDecoding()),
         readerTimezone(getTimezoneByName(opts.getTimezoneName())),
         filter(_filter),
@@ -282,28 +305,31 @@ namespace orc {
     rowsInCurrentStripe = 0;
     numRowGroupsInStripeRange = 0;
     useTightNumericVector = opts.getUseTightNumericVector();
-    uint64_t rowTotal = 0;
     rowTotalInRange = 0;
 
-    firstRowOfStripe.resize(numberOfStripes);
-    for (size_t i = 0; i < numberOfStripes; ++i) {
-      firstRowOfStripe[i] = rowTotal;
-      proto::StripeInformation stripeInfo = 
footer->stripes(static_cast<int>(i));
-      rowTotal += stripeInfo.numberofrows();
-      bool isStripeInRange = stripeInfo.offset() >= opts.getOffset() &&
-                             stripeInfo.offset() < opts.getOffset() + 
opts.getLength();
-      if (isStripeInRange) {
-        rowTotalInRange += stripeInfo.numberofrows();
-        if (i < currentStripe) {
-          currentStripe = i;
-        }
-        if (i >= lastStripe) {
-          lastStripe = i + 1;
-        }
-        if (footer->rowindexstride() > 0) {
-          numRowGroupsInStripeRange +=
-              (stripeInfo.numberofrows() + footer->rowindexstride() - 1) / 
footer->rowindexstride();
-        }
+    // Stripe offsets are ordered. For example, a one-stripe child of an 
eight-stripe
+    // source reuses the eight absolute prefixes and visits only its own 
directory entry.
+    const auto& stripes = footer->stripes();
+    const auto offsetLess = [](const proto::StripeInformation& stripe, 
uint64_t offset) {
+      return stripe.offset() < offset;
+    };
+    const auto begin =
+        std::lower_bound(stripes.begin(), stripes.end(), opts.getOffset(), 
offsetLess);
+    const auto end =
+        std::lower_bound(begin, stripes.end(), opts.getOffset() + 
opts.getLength(), offsetLess);
+    if (begin != end) {
+      currentStripe = static_cast<uint64_t>(begin - stripes.begin());
+      lastStripe = static_cast<uint64_t>(end - stripes.begin());
+    }
+    if (contents->readerMetrics != nullptr) {
+      contents->readerMetrics->RowReaderStripeVisitCount.fetch_add(
+          static_cast<uint64_t>(end - begin), std::memory_order_relaxed);
+    }
+    for (auto stripe = begin; stripe != end; ++stripe) {
+      rowTotalInRange += stripe->numberofrows();
+      if (footer->rowindexstride() > 0) {
+        numRowGroupsInStripeRange +=
+            (stripe->numberofrows() + footer->rowindexstride() - 1) / 
footer->rowindexstride();
       }
     }
     firstStripe = currentStripe;
@@ -556,10 +582,11 @@ namespace orc {
       return;
     }
 
-    uint64_t seekToStripe = 0;
-    while (seekToStripe + 1 < lastStripe && firstRowOfStripe[seekToStripe + 1] 
<= rowNumber) {
-      seekToStripe++;
-    }
+    // Absolute prefixes are shared with all physical children. A seek in a 
late child
+    // must not walk every preceding sibling's prefix again.
+    uint64_t seekToStripe = static_cast<uint64_t>(
+        std::upper_bound(firstRowOfStripe.data(), firstRowOfStripe.data() + 
lastStripe, rowNumber) -
+        firstRowOfStripe.data() - 1);
 
     // seeking before the first stripe
     if (seekToStripe < firstStripe) {
@@ -1783,52 +1810,73 @@ namespace orc {
     contents->pool = options.getMemoryPool();
     contents->errorStream = options.getErrorStream();
     contents->readerMetrics = options.getReaderMetrics();
-    std::string serializedFooter = options.getSerializedFileTail();
     uint64_t fileLength;
     uint64_t postscriptLength;
-    if (serializedFooter.length() != 0) {
-      // Parse the file tail from the serialized one.
-      proto::FileTail tail;
-      if (!tail.ParseFromString(serializedFooter)) {
-        throw ParseError("Failed to parse the file tail from string");
-      }
-      contents->postscript = 
std::make_unique<proto::PostScript>(tail.postscript());
-      contents->footer = std::make_unique<proto::Footer>(tail.footer());
-      fileLength = tail.filelength();
-      postscriptLength = tail.postscriptlength();
+    contents->fileTail = options.getSharedFileTail();
+    // Physical siblings already have a parsed tail. Re-parsing its 
serialization would
+    // repeat file-wide protobuf allocation and prefix construction for every 
child.
+    if (contents->fileTail) {
+      contents->postscript = contents->fileTail->postscript;
+      contents->footer = contents->fileTail->footer;
+      fileLength = contents->fileTail->fileLength;
+      postscriptLength = contents->fileTail->postscriptLength;
     } else {
-      // figure out the size of the file using the option or filesystem
-      fileLength = std::min(options.getTailLocation(), 
static_cast<uint64_t>(stream->getLength()));
+      std::string serializedFooter = options.getSerializedFileTail();
+      if (serializedFooter.length() != 0) {
+        // Parse the file tail from the serialized one.
+        proto::FileTail tail;
+        if (!tail.ParseFromString(serializedFooter)) {
+          throw ParseError("Failed to parse the file tail from string");
+        }
+        contents->postscript = 
std::make_unique<proto::PostScript>(tail.postscript());
+        contents->footer = std::make_unique<proto::Footer>(tail.footer());
+        fileLength = tail.filelength();
+        postscriptLength = tail.postscriptlength();
+      } else {
+        // figure out the size of the file using the option or filesystem
+        fileLength =
+            std::min(options.getTailLocation(), 
static_cast<uint64_t>(stream->getLength()));
+
+        // read last bytes into buffer to get PostScript
+        uint64_t readSize = std::min(fileLength, DIRECTORY_SIZE_GUESS);
+        if (readSize < 4) {
+          throw ParseError("File size too small");
+        }
+        auto buffer = std::make_unique<DataBuffer<char>>(*contents->pool, 
readSize);
+        stream->read(buffer->data(), readSize, fileLength - readSize);
+
+        postscriptLength = buffer->data()[readSize - 1] & 0xff;
+        contents->postscript = readPostscript(stream.get(), buffer.get(), 
postscriptLength);
+        uint64_t footerSize = contents->postscript->footerlength();
+        uint64_t tailSize = 1 + postscriptLength + footerSize;
+        if (tailSize >= fileLength) {
+          std::stringstream msg;
+          msg << "Invalid ORC tailSize=" << tailSize << ", fileLength=" << 
fileLength;
+          throw ParseError(msg.str());
+        }
+        uint64_t footerOffset;
 
-      // read last bytes into buffer to get PostScript
-      uint64_t readSize = std::min(fileLength, DIRECTORY_SIZE_GUESS);
-      if (readSize < 4) {
-        throw ParseError("File size too small");
-      }
-      auto buffer = std::make_unique<DataBuffer<char>>(*contents->pool, 
readSize);
-      stream->read(buffer->data(), readSize, fileLength - readSize);
-
-      postscriptLength = buffer->data()[readSize - 1] & 0xff;
-      contents->postscript = readPostscript(stream.get(), buffer.get(), 
postscriptLength);
-      uint64_t footerSize = contents->postscript->footerlength();
-      uint64_t tailSize = 1 + postscriptLength + footerSize;
-      if (tailSize >= fileLength) {
-        std::stringstream msg;
-        msg << "Invalid ORC tailSize=" << tailSize << ", fileLength=" << 
fileLength;
-        throw ParseError(msg.str());
-      }
-      uint64_t footerOffset;
+        if (tailSize > readSize) {
+          buffer->resize(footerSize);
+          stream->read(buffer->data(), footerSize, fileLength - tailSize);
+          footerOffset = 0;
+        } else {
+          footerOffset = readSize - tailSize;
+        }
 
-      if (tailSize > readSize) {
-        buffer->resize(footerSize);
-        stream->read(buffer->data(), footerSize, fileLength - tailSize);
-        footerOffset = 0;
-      } else {
-        footerOffset = readSize - tailSize;
+        contents->footer =
+            readFooter(stream.get(), buffer.get(), footerOffset, 
*contents->postscript,
+                       *contents->pool, contents->readerMetrics);
       }
-
-      contents->footer = readFooter(stream.get(), buffer.get(), footerOffset, 
*contents->postscript,
-                                    *contents->pool, contents->readerMetrics);
+      if (contents->readerMetrics != nullptr) {
+        contents->readerMetrics->FileTailParseCount.fetch_add(1, 
std::memory_order_relaxed);
+      }
+      auto tail = std::make_shared<SharedFileTail>(*contents->pool);
+      tail->postscript = contents->postscript;
+      tail->footer = contents->footer;
+      tail->fileLength = fileLength;
+      tail->postscriptLength = postscriptLength;
+      contents->fileTail = std::move(tail);
     }
     contents->isDecimalAsLong = false;
     if (contents->postscript->version_size() == 2) {
diff --git a/c++/src/Reader.hh b/c++/src/Reader.hh
index 537a3a5908a..94b1e5b31ba 100644
--- a/c++/src/Reader.hh
+++ b/c++/src/Reader.hh
@@ -95,11 +95,32 @@ namespace orc {
     std::shared_ptr<const proto::Metadata> metadata;
   };
 
+  // The parsed tail is immutable after publication. Prefixes are built once 
on first RowReader
+  // use, so metadata-only readers do not allocate them. The source allocator 
must outlive this
+  // handle; no reader, stream, mutable schema or metrics are retained.
+  class SharedFileTail {
+   public:
+    explicit SharedFileTail(MemoryPool& memoryPool) : pool(memoryPool) {}
+
+    const DataBuffer<uint64_t>& getFirstRowOfStripe(ReaderMetrics* metrics) 
const;
+
+    std::shared_ptr<const proto::PostScript> postscript;
+    std::shared_ptr<const proto::Footer> footer;
+    uint64_t fileLength;
+    uint64_t postscriptLength;
+
+   private:
+    MemoryPool& pool;
+    mutable std::once_flag prefixOnce;
+    mutable std::unique_ptr<DataBuffer<uint64_t>> firstRowOfStripe;
+  };
+
   /** State shared between one Reader and its RowReaders. */
   struct FileContents {
     std::unique_ptr<InputStream> stream;
-    std::unique_ptr<proto::PostScript> postscript;
-    std::unique_ptr<proto::Footer> footer;
+    std::shared_ptr<const proto::PostScript> postscript;
+    std::shared_ptr<const proto::Footer> footer;
+    std::shared_ptr<const SharedFileTail> fileTail;
     std::unique_ptr<Type> schema;
     uint64_t blockSize;
     CompressionKind compression;
@@ -181,8 +202,8 @@ namespace orc {
     std::vector<bool> selectedColumns;
 
     // footer
-    proto::Footer* footer;
-    DataBuffer<uint64_t> firstRowOfStripe;
+    const proto::Footer* footer;
+    const DataBuffer<uint64_t>& firstRowOfStripe;
     mutable std::unique_ptr<Type> selectedSchema;
     bool skipBloomFilters;
 
@@ -325,7 +346,7 @@ namespace orc {
     const uint64_t postscriptLength;
 
     // footer
-    proto::Footer* footer;
+    const proto::Footer* footer;
     uint64_t numberOfStripes;
     std::unique_ptr<SargsApplier> sargsApplier;
     std::vector<int> getNeedReadStripes(const RowReaderOptions& opts) override;
@@ -411,6 +432,10 @@ namespace orc {
 
     std::string getSerializedFileTail() const override;
 
+    std::shared_ptr<const SharedFileTail> getSharedFileTail() const override {
+      return contents->fileTail;
+    }
+
     const Type& getType() const override;
 
     bool hasCorrectStatistics() const override;
diff --git a/c++/test/TestReader.cc b/c++/test/TestReader.cc
index f709f693f10..33a307035cd 100644
--- a/c++/test/TestReader.cc
+++ b/c++/test/TestReader.cc
@@ -17,6 +17,7 @@
  */
 
 #include <cstring>
+#include <thread>
 
 #include "Reader.hh"
 #include "orc/Reader.hh"
@@ -34,6 +35,154 @@ namespace orc {
 
   static const int DEFAULT_MEM_STREAM_SIZE = 1024 * 1024;  // 1M
 
+  TEST(TestReader, SharedFileTailKeepsIndependentRangeReaders) {
+    constexpr uint64_t STRIPES = 8;
+    // Exceed the RLE encoder's buffer so every add() has encoded bytes to 
trigger a stripe flush.
+    constexpr uint64_t ROWS = 3000;
+    MemoryOutputStream stream(DEFAULT_MEM_STREAM_SIZE);
+    auto type = Type::buildTypeFromString("struct<id:bigint>");
+    WriterOptions writeOptions;
+    
writeOptions.setStripeSize(1).setCompression(CompressionKind_NONE).setRowIndexStride(0);
+    auto writer = createWriter(*type, &stream, writeOptions);
+    auto batch = writer->createRowBatch(ROWS);
+    auto& root = dynamic_cast<StructVectorBatch&>(*batch);
+    auto& ids = dynamic_cast<LongVectorBatch&>(*root.fields[0]);
+    for (uint64_t stripe = 0; stripe < STRIPES; ++stripe) {
+      for (uint64_t row = 0; row < ROWS; ++row) {
+        ids.data[row] = static_cast<int64_t>(stripe * ROWS + row);
+      }
+      root.numElements = ids.numElements = ROWS;
+      writer->add(*batch);
+    }
+    writer->close();
+
+    std::shared_ptr<const SharedFileTail> shared;
+    std::vector<uint64_t> offsets;
+    std::string serialized;
+    // No parent stream, metrics or schema survives this scope.
+    {
+      ReaderMetrics metrics;
+      ReaderOptions options;
+      options.setReaderMetrics(&metrics);
+      auto parent = createReader(
+          std::make_unique<MemoryInputStream>(stream.getData(), 
stream.getLength()), options);
+      ASSERT_EQ(STRIPES, parent->getNumberOfStripes());
+      shared = parent->getSharedFileTail();
+      serialized = parent->getSerializedFileTail();
+      for (uint64_t stripe = 0; stripe < STRIPES; ++stripe) {
+        offsets.push_back(parent->getStripe(stripe)->getOffset());
+      }
+      EXPECT_EQ(1, metrics.FileTailParseCount.load());
+      EXPECT_EQ(0, metrics.StripeDirectoryEntryCount.load());
+    }
+    ASSERT_NE(nullptr, shared);
+    offsets.push_back(stream.getLength());
+
+    // Independent children race on immutable metadata, first individually and 
then coalesced.
+    for (uint64_t width : {uint64_t{1}, uint64_t{4}}) {
+      std::atomic<uint64_t> ready{0};
+      std::atomic<bool> start{false};
+      std::atomic<uint64_t> prefixEntries{0};
+      std::vector<std::thread> children;
+      for (uint64_t first = 0; first < STRIPES; first += width) {
+        children.emplace_back([&, first] {
+          ready.fetch_add(1, std::memory_order_release);
+          while (!start.load(std::memory_order_acquire)) {
+            std::this_thread::yield();
+          }
+          ReaderMetrics metrics;
+          const ReaderOptions options = ReaderOptions()
+                                            .setSharedFileTail(shared)
+                                            .setReaderMetrics(&metrics)
+                                            .setSerializedFileTail("not a 
serialized tail");
+          auto reader = createReader(
+              std::make_unique<MemoryInputStream>(stream.getData(), 
stream.getLength()), options);
+          EXPECT_EQ(shared, reader->getSharedFileTail());
+          auto rows = reader->createRowReader(
+              RowReaderOptions().range(offsets[first], offsets[first + width] 
- offsets[first]));
+          EXPECT_EQ(width * ROWS, rows->getNumberOfRows());
+          EXPECT_EQ(0, metrics.FileTailParseCount.load());
+          EXPECT_EQ(width, metrics.RowReaderStripeVisitCount.load());
+          auto data = rows->createRowBatch(37);
+          uint64_t expected = first * ROWS;
+          while (rows->next(*data)) {
+            EXPECT_EQ(expected, rows->getRowNumber());
+            auto& values =
+                
dynamic_cast<LongVectorBatch&>(*dynamic_cast<StructVectorBatch&>(*data).fields[0]);
+            for (uint64_t row = 0; row < data->numElements; ++row) {
+              EXPECT_EQ(static_cast<int64_t>(expected++), values.data[row]);
+            }
+          }
+          EXPECT_EQ((first + width) * ROWS, expected);
+          const auto seekRow = (first + width) * ROWS - 1;
+          rows->seekToRow(seekRow);
+          ASSERT_TRUE(rows->next(*data));
+          EXPECT_EQ(seekRow, rows->getRowNumber());
+          EXPECT_EQ(1, data->numElements);
+          auto& values =
+              
dynamic_cast<LongVectorBatch&>(*dynamic_cast<StructVectorBatch&>(*data).fields[0]);
+          EXPECT_EQ(static_cast<int64_t>(seekRow), values.data[0]);
+          prefixEntries.fetch_add(metrics.StripeDirectoryEntryCount.load(),
+                                  std::memory_order_relaxed);
+        });
+      }
+      while (ready.load(std::memory_order_acquire) != STRIPES / width) {
+        std::this_thread::yield();
+      }
+      start.store(true, std::memory_order_release);
+      for (auto& child : children) {
+        child.join();
+      }
+      EXPECT_EQ(width == 1 ? STRIPES : 0, prefixEntries.load());
+    }
+
+    // The existing serialized-tail API still builds an independent parsed 
context.
+    ReaderMetrics metrics;
+    const ReaderOptions options =
+        
ReaderOptions().setSerializedFileTail(serialized).setReaderMetrics(&metrics);
+    auto reader = createReader(
+        std::make_unique<MemoryInputStream>(stream.getData(), 
stream.getLength()), options);
+    EXPECT_NE(shared, reader->getSharedFileTail());
+    EXPECT_EQ(1, metrics.FileTailParseCount.load());
+    EXPECT_EQ(0, metrics.StripeDirectoryEntryCount.load());
+    EXPECT_EQ(STRIPES * ROWS, reader->getNumberOfRows());
+    EXPECT_EQ(0, metrics.StripeDirectoryEntryCount.load());
+    auto empty = reader->createRowReader(RowReaderOptions().range(offsets[2], 
0));
+    EXPECT_EQ(STRIPES, metrics.StripeDirectoryEntryCount.load());
+    EXPECT_EQ(0, empty->getNumberOfRows());
+    EXPECT_EQ(0, metrics.RowReaderStripeVisitCount.load());
+    EXPECT_FALSE(empty->next(*empty->createRowBatch(1)));
+
+    proto::FileTail corrupt;
+    ASSERT_TRUE(corrupt.ParseFromString(serialized));
+    corrupt.mutable_footer()->mutable_stripes(1)->set_offset(offsets[0] - 1);
+    const ReaderOptions corruptOptions =
+        ReaderOptions().setSerializedFileTail(corrupt.SerializeAsString());
+    auto corruptReader = createReader(
+        std::make_unique<MemoryInputStream>(stream.getData(), 
stream.getLength()), corruptOptions);
+    EXPECT_THROW(corruptReader->createRowReader(), ParseError);
+    EXPECT_THROW(corruptReader->createRowReader(), ParseError);
+  }
+
+  TEST(TestReader, SharedFileTailSupportsEmptyFiles) {
+    MemoryOutputStream stream(DEFAULT_MEM_STREAM_SIZE);
+    auto type = Type::buildTypeFromString("struct<id:bigint>");
+    auto writer = createWriter(*type, &stream, WriterOptions());
+    writer->close();
+    auto parent = createReader(
+        std::make_unique<MemoryInputStream>(stream.getData(), 
stream.getLength()), ReaderOptions());
+    const ReaderOptions options = 
ReaderOptions().setSharedFileTail(parent->getSharedFileTail());
+    parent.reset();
+    auto reader = createReader(
+        std::make_unique<MemoryInputStream>(stream.getData(), 
stream.getLength()), options);
+    auto rows = reader->createRowReader();
+    EXPECT_EQ(0, rows->getNumberOfRows());
+    auto batch = rows->createRowBatch(1);
+    EXPECT_FALSE(rows->next(*batch));
+    rows->seekToRow(0);
+    EXPECT_FALSE(rows->next(*batch));
+  }
+
   TEST(TestReader, testWriterVersions) {
     EXPECT_EQ("original", writerVersionToString(WriterVersion_ORIGINAL));
     EXPECT_EQ("HIVE-8732", writerVersionToString(WriterVersion_HIVE_8732));


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to