morningman opened a new pull request, #436:
URL: https://github.com/apache/doris-thirdparty/pull/436
### Background
Independent ORC range readers have separate streams, schemas and predicates,
but need the same file footer and absolute stripe row positions. Doris physical
splitting creates multiple such readers for one immutable file. Shared stripe
statistics were added in #435; serialized file tails still require each child
to parse the footer and rebuild the row-prefix directory.
### What problem does this PR solve?
Issue Number: None
Related PR: #435 (shared stripe statistics), apache/doris#70664 (downstream
physical stripe readers)
PR Type: Performance
Problem Summary:
Physical stripe children repeat file-wide tail parsing, prefix construction
and range-directory traversal. Share an immutable parsed tail and one lazily
initialized row-prefix buffer, then locate each reader's byte and row
boundaries with binary searches. Metadata-only opens remain lazy, while
streams, mutable schemas, predicates and metrics stay private. Existing
serialized-tail callers remain supported.
### Details
- **Change map:** Five files, +318/-70: production code +169/-70 for the
shared handle, lazy prefix initialization, bounded range/seek lookup and
counters; tests +149 for concurrency, lifetime and edge cases. The second
commit preserves lazy metadata-only opens.
- **Why it was slow:** `createReader` reparses the serialized tail for each
child. Each `RowReader` also rebuilds all absolute stripe prefixes and scans
the file-wide stripe directory, so setup grows with both child count and stripe
count.
- **Optimization:** Reuse immutable Footer/PostScript data through
`SharedFileTail`. Build the source-pool-backed prefix once on the first row
read, publish it only after successful initialization, and permit retries after
exceptions. Binary-search range boundaries and accumulate only the owned
interval. Validate stripe ordering before using binary search. Add counters for
tail parsing, prefix entries and range-accumulation visits.
- **Results:** In a local macOS ASAN Doris fixture with eight stripes, a
planning reader and eight one-stripe children, the candidate measured the
counts below. The comparison temporarily restored repeated setup in the same
candidate; it was not an execution of an untouched historical commit. The
restored path failed nine expected counter assertions while row-value
assertions still passed. These are operation counts; no elapsed-time speedup is
claimed.
| Work | Restored repeated setup | Candidate |
| --- | ---: | ---: |
| File-tail parses | 9 | 1 |
| Stripe-prefix entries built | 72 | 8 |
| Range-accumulation stripe visits | 72 | 16 |
- **Applicability and cost:** Sharing helps sibling readers of the same
immutable file. The first row reader builds one file-wide prefix; metadata-only
opens build none. Binary-search comparisons are additional logarithmic work
beyond the counted visits. The source memory pool must outlive all shared
handles. The added reader interface requires native consumers to rebuild; file
formats remain unchanged. No RELEASE timing benchmark was run.
### Testing
Locally, Doris `run-be-ut.sh` passed 255 selected tests under ASAN, with
zero failures, errors or skips, using the exact SDK source in this PR and the
pending Doris integration. This includes 58 SDK tests across reader,
row-reader, read-intent, predicate and stripe-statistics suites, plus 197 Doris
ORC tests.
New SDK coverage checks cold concurrent readers after parent destruction,
single/coalesced intervals, one shared prefix build, zero metadata-only prefix
work, warm reuse, values and absolute row positions, seeks, serialized-tail
compatibility, empty files/ranges and repeated malformed-directory failures.
The downstream counter test detects the controlled restoration described above;
the candidate was restored and its source hashes verified before the passing
run.
### Notes for Reviewers
The published head of apache/doris#70664 does not yet consume
`SharedFileTail`. Its local follow-up passes the parsed handle to children and
supplies the integration counters used above. This SDK PR contains only the two
commits after merged #435 (`4ec2590..89c98c2`).
### Release note
None
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No; SDK lifetime and sharing contracts are
documented in the header.
- Pre-submission status: Partially verified: local ASAN tests passed; no
RELEASE timing benchmark or external cluster regression was run.
### AI Disclosure
- Tool: Codex; GPT-6.1-Sol for review
- Generated: SDK implementation, tests and this description.
- Human review: not stated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]