bobhan1 opened a new pull request, #438:
URL: https://github.com/apache/doris-thirdparty/pull/438

   ### Background
   
   Index compaction translates source document IDs into destination 
index/document pairs. Its dense matrix parameter is copied into IndexWriter, 
preventing callers from supplying bounded compressed mappings such as the 
storage introduced by apache/doris#70350.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: apache/doris#70350 (compressed storage), apache/doris#69313 and 
#420 (existing token-statistics prerequisite)
   PR Type: Performance
   
   Problem Summary:
   
   Add a caller-owned `DocIdMapping` interface and an `indexCompaction` 
overload that borrows it during synchronous compaction. Postings, both norms 
paths and null bitmaps use fallible point lookups, without depending on Doris 
headers or materializing a dense matrix. The existing dense overload and 
`setTransVec` remain available through an owned dense adapter.
   
   ### Details
   
   - **Change map:** 253 production lines changed and 160 test lines across 
four files; no generated files or new dependencies.
   - **Why it was slow:** The old API requires a full matrix and copies its row 
pairs into the writer. The matrix alone is 8 bytes per physical source row, 
before the extra copy and other compaction allocations.
   - **Optimization:** Borrow a provider for the new entry. Terms may revisit 
lower source doc IDs; directory positions and the existing deleted-row sentinel 
are preserved. The borrowed pointer resets on success and exception. Diagnostic 
source locations come from the queued posting rather than a reverse scan of the 
matrix.
   - **Behavior change:** Successful results and the legacy dense entry remain 
compatible. Lookup failures abort compaction and release the resources acquired 
by the new provider path.
   - **Failure handling:** A lookup error propagates as `CLuceneError`, never 
as a deleted document. ASAN failure injection exposed leaks after output 
creation and inside posting merge. RAII now owns queued/active source 
iterators, destination writers and segment metadata; norms/null outputs join 
the existing finally cleanup. All three lookup-failure phases are covered.
   - **Results:** Independent ASAN/LSAN compaction tests pass 9/9. The 
coordinated local Doris stack passes 67/67 targeted ASAN tests and a complete 
PCH-disabled production Release BE build. These are correctness/build results; 
no optimized throughput or billion-row compaction measurements were repeated 
for this change.
   - **Applicability and cost:** One virtual call per lookup; a document can be 
revisited by different terms or fields. Representation and cache policy belong 
to the caller. The legacy dense entry retains its ownership/copying behavior; 
clients adopt the new entry to avoid the copy. IndexWriter layout changes 
require clients to rebuild.
   
   ### Testing
   
   Independent CLucene build with Clang 20.1.8 and `CMAKE_BUILD_TYPE=ASAN`, 
using the repository's CMake target:
   
   ```sh
   cmake --build <build-dir> --target cl_test --parallel 100
   cd <build-dir>/bin
   ./cl_test -t IndexCompaction
   ```
   
   All nine tests pass with ASAN/LSAN and no new suppressions: seven existing 
cases, borrowed/dense equivalence for postings/positions, norm bytes, token 
totals and null row IDs, source rewinds, diagnostic output, and lookup failures 
in norms, postings and null-only remapping. The all-phase failure test leaks 
with the earlier raw ownership and passes after the cleanup fix.
   
   The local Doris integration uses master 
`a02ea2b6572e51fc19988f268c1a6fe1274d7286`, the production eligibility hunk 
from the still-open apache/doris#69313, and the new Doris adapter. That 
prerequisite is required by the norms/frequency guard already merged in #420; 
it is not part of this interface PR. No Doris adoption PR has been opened yet.
   
   ```sh
   bash run-be-ut.sh --run 
--filter='CluceneRowIdMappingTest.*:*TestRowIdConversion*:CompressedRowIdMapTest.*:IndexCompactionTest.*:IndexCompactionWriteIndexTest.*:SniiRowIdConversionTest.*-IndexCompactionTest.tes_wikipedia*'
 -j100
   ENABLE_PCH=OFF ENABLE_UNITY_BUILD=ON bash build.sh --be --coverage -j100
   ```
   
   Doris tests cover bounded cross-block/backward lookups, malformed input and 
lookup errors, dense/compressed in-memory index equivalence, and real Duplicate 
Key/MoW compaction UTs that verify COMPRESSED mode and compare against raw 
rebuilds. The production build passes compilation, linking, installation and 
packaging with `MAKE_TEST=OFF`; the packaged binary meets the GLIBC 2.17 
baseline. clang-format 16, build hygiene and whitespace checks pass. ASAN 
lookup timing output is diagnostic only. The final targeted run includes both 
compressed-mode pipeline tests and representative legacy cases; the ten long 
legacy wiki variants are excluded from that final rerun after correcting the 
test helper to call production mapping selection/finalization.
   
   ### Release note
   
   Index compaction can borrow a document-ID mapping provider without requiring 
a dense matrix.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No; interface ownership and lookup semantics 
are documented in source.
   - Pre-submission status: Ready
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex (GPT-6)
   - Generated: Implementation, tests and description.
   - Human review: Pending.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to