airborne12 opened a new pull request, #69365: URL: https://github.com/apache/doris/pull/69365
### What problem does this PR solve? Issue Number: N/A Related PR: #67977 Problem Summary: Backport [#67977](https://github.com/apache/doris/pull/67977), merged squash `3c6af92291b14979298a25b02b1abc264287f6f4`, to branch-4.2. The `index_disk_usage` table function reports each inverted or ANN index's physical bytes and components at tablet, rowset, or segment level. The source author, cherry-pick provenance, scan limits, replica routing, visible-version capture, query IO accounting, test assertions, and regression output are preserved. Adaptations for branch-4.2: - Override `GenericReader::get_next_block`, keep direct `init_reader`, and use `SlotDescriptor::get_data_type_ptr`. This branch has no `ReaderInitContext` adapter. - Traverse existing rowset segment IDs and paths, and read index-file metadata and persisted row counts from `RowsetMeta`. Missing row counts still load segment footers directly with the query IO context, without caching a transient row-count failure. - Use `IndexDef.IndexType` and `VariableMgr.VarAttr`. Both description slots required by the target documentation generator contain English text. - Select the VARIANT fixture parser using the target branch's `enable_variant_v2` convention, retaining the same JSON documents, typed path, queries, and expected output. - Use the existing `TabletSchema::create_block` API for the two-column integer rowset fixture. Its columns are unchanged, and all test assertions are preserved. - Preserve existing Hudi/Lance TVFs, scanner defaults, and session-clone tests in neighboring code. Use equivalent ranges algorithms and explicit `Status` returns to satisfy static checks. - Omit only the source's Storage unity skip: branch-4.2 already compiles Storage sources separately. The source PR has no linked issue; no revert or follow-up fix was found during source resolution. The new optional Thrift fields and enum value retain their source wire IDs, checked against maintained branches. No persisted index format is changed. ### Release note Add `index_disk_usage` to report the physical size and components of each index. ### Check List (For Author) Local validation on `e440788164dd07ba417d97dfae1a0719b7ccacdc`: - Standard `./build.sh --be -j48` and `./build.sh --fe -j48`: PASS. BE uses ASAN with `GLIBC_COMPATIBILITY=ON`; FE uses JDK 17. The final regression-fixture adaptation leaves all production and unit-test inputs identical to the tested e369dd033a3 candidate. - `./run-be-ut.sh --run --filter='IndexDiskUsage*:SniiSegmentReaderLogicalIndexesTest.*' -j48`: 55/55 tests passed in 8 suites. The same binary first listed the tests; the executed names and XML exactly match all original feature tests. No sanitizer report. - `./run-fe-ut.sh --run 'org.apache.doris.datasource.tvf.source.IndexDiskUsageScanNodeTest,org.apache.doris.tablefunction.IndexDiskUsageTableValuedFunctionTest,org.apache.doris.qe.SessionVariablesTest'`: 53/53 passed (14 + 23 + 16), with no failures, errors, or skips. - `./run-regression-test.sh --conf <isolated-cluster-config> --run -d external_table_p0/tvf -s test_index_disk_usage_tvf`: PASS against a private single-FE/BE ASAN cluster. V2, V3, SNII, component totals, all levels, position details, filters, rollup/VARIANT labels, and error cases use the original suite and expected output. - Full CI formatting scope on the merged snapshot with branch-4.2 `2134b43d9c0f2129823a131ebd7499d659ccf6c2`: clang-format 16 and JDK 17 `mvn clean checkstyle:check` across all 28 FE reactor modules passed. The committed PR English check passed for all 29 source files, with no exceptions. - Configured clang-tidy checks passed on PR lines in all 5 modified production and 5 test translation units. Static-check baseline qualification: unchanged `be/src/core/types.h:578` contains an unmatched `NOLINTEND` that stops the installed clang-tidy. An analysis-only VFS overlay removes exactly that comment; the real header and checker configuration remain unchanged, no checks are disabled, and the original/overlay preprocessed 25 MB production translation units are byte-identical. The raw baseline error, exact overlay, token-identity proof, and per-file results are retained locally. Local runtime configuration: the existing startup `SKIP_CHECK_ULIMIT=true` switch was used for this small isolated regression because the shared host's `vm.max_map_count` is 655360. The BE had 7154 mappings in the recorded post-test snapshot. No host settings or sanitizer settings were changed. These are targeted feature tests; the full BE/FE runtime suites are left to CI. - Test: - [x] Regression test: `external_table_p0/tvf/test_index_disk_usage_tvf`. - [x] Unit Test: the original feature BE/FE tests and related session-variable tests. - [ ] Manual test. - [ ] No need to test. - Behavior changed: - [x] Yes. Add the table function to branch-4.2. - Does this need documentation? - [x] Yes. Same documentation requirement as #67977; no separate website PR is linked by the source. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note. - [ ] Confirm test cases. - [ ] Confirm documentation. - [ ] Add branch pick label. <details> <summary>Source hunk audit</summary> | Source path | Source hunk | Result | Target evidence / adaptation | |---|---|---|---| | `be/src/exec/scan/meta_scanner.cpp` | `@@ -37,6 +37,7 @@` | Ported | Keep target-branch context; preserve the source change. | | `be/src/exec/scan/meta_scanner.cpp` | `@@ -69,6 +70,11 @@ Status MetaScanner::_open_impl(RuntimeState* state) {` | Ported | Keep target-branch context; preserve the source change. | | `be/src/format/table/index_disk_usage_reader.cpp` | `@@ -0,0 +1,436 @@` | Adapted | Override GenericReader::get_next_block; use SlotDescriptor::get_data_type_ptr, which returns the target branch DataTypePtr. All collection and output logic is preserved. | | `be/src/format/table/index_disk_usage_reader.h` | `@@ -0,0 +1,121 @@` | Adapted | Declare the branch-4.2 get_next_block virtual entry point. Remove the master-only ReaderInitContext adapter; MetaScanner still calls init_reader directly. | | `be/src/storage/CMakeLists.txt` | `@@ -56,6 +56,9 @@ pch_reuse(Storage)` | N-A | branch-4.2 Storage CMake ends after pch_reuse(Storage) and has no storage unity batches or STORAGE_UNITY_SKIP; all sources already compile separately. | | `be/src/storage/index/index_disk_usage.cpp` | `@@ -0,0 +1,581 @@` | Adapted | Use branch-4.2 contiguous segment IDs, Rowset::segment_path, RowsetMeta index-file metadata and persisted row counts; load missing row counts directly through BetaRowset::load_segment without populating the rowset row-count cache. Use equivalent ranges algorithms and explicit Status error returns to satisfy branch-4.2 static checks without changing skipped/missing-container outcomes. | | `be/src/storage/index/index_disk_usage.h` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) | | | `be/src/storage/index/index_file_reader.cpp` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) | | | `be/src/storage/index/index_file_reader.h` | `@@ -95,6 +95,16 @@ public:` | Ported | Keep target-branch context; preserve the source change. | | `be/src/storage/index/inverted/snii/reader/snii_segment_reader.cpp` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) | | | `be/src/storage/index/inverted/snii/reader/snii_segment_reader.h` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) | | | `be/src/storage/segment/page_decoder.h` | `@@ -20,6 +20,7 @@` | Ported | Keep target-branch context; preserve the source change. | | `be/test/format/table/index_disk_usage_reader_test.cpp` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) | | | `be/test/storage/index/index_disk_usage_aggregate_test.cpp` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) | | | `be/test/storage/index/index_disk_usage_rowset_test.cpp` | `@@ -0,0 +1,249 @@` | Adapted | Use branch-4.2 TabletSchema::create_block for the two-column integer storage fixture. This produces the same columns because the fixture has no dropped or pruned columns. Preserve all original test assertions. | | `be/test/storage/index/index_disk_usage_test.cpp` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) | | | `be/test/storage/index/inverted/snii/reader/snii_segment_reader_logical_indexes_test.cpp` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) | | | `fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java` | `@@ -28,6 +28,7 @@ import org.apache.doris.nereids.trees.expressions.functions.table.GroupCommit;` | Ported | Keep target-branch context; preserve the source change. | | `fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java` | `@@ -65,6 +66,7 @@ public class BuiltinTableValuedFunctions implements FunctionHelper {` | Ported | Keep target-branch context; preserve the source change. | | `fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/IndexDiskUsageScanNode.java` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) | | | `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/IndexDiskUsage.java` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) | | | `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/TableValuedFunctionVisitor.java` | `@@ -28,6 +28,7 @@ import org.apache.doris.nereids.trees.expressions.functions.table.GroupCommit;` | Ported | Keep target-branch context; preserve the source change. | | `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/TableValuedFunctionVisitor.java` | `@@ -68,6 +69,10 @@ public interface TableValuedFunctionVisitor<R, C> {` | Ported | Keep target-branch context; preserve the source change. | | `fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java` | `@@ -1154,9 +1154,8 @@ public class OlapScanNode extends ScanNode {` | Ported | Keep target-branch context; preserve the source change. | | `fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java` | `@@ -95,6 +95,9 @@ public class SessionVariable implements Serializable, Writable {` | Ported | Keep target-branch context; preserve the source change. | | `fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java` | `@@ -1172,6 +1175,15 @@ public class SessionVariable implements Serializable, Writable {` | Adapted | Use VariableMgr.VarAttr and both description slots required by branch-4.2 DocGenerator; keep both limits, defaults and needForward=true. Existing scanner settings remain unchanged. | | `fe/fe-core/src/main/java/org/apache/doris/tablefunction/IndexDiskUsageTableValuedFunction.java` | `@@ -0,0 +1,478 @@` | Adapted | Use branch-4.2 IndexDef.IndexType, which Index.getIndexType returns; preserve the source enum comparisons and all test assertions. | | `fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java` | `@@ -69,6 +69,8 @@ public abstract class TableValuedFunctionIf {` | Ported | Keep target-branch context; preserve the source change. | | `fe/fe-core/src/test/java/org/apache/doris/datasource/tvf/source/IndexDiskUsageScanNodeTest.java` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) | | | `fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java` | `@@ -79,6 +79,23 @@ public class SessionVariablesTest extends TestWithFeService {` | Ported | Keep target-branch context; preserve the source change. | | `fe/fe-core/src/test/java/org/apache/doris/tablefunction/IndexDiskUsageTableValuedFunctionTest.java` | `@@ -0,0 +1,489 @@` | Adapted | Use branch-4.2 IndexDef.IndexType, which Index.getIndexType returns; preserve the source enum comparisons and all test assertions. | | `gensrc/thrift/PlanNodes.thrift` | `@@ -839,6 +839,29 @@ struct TLanceIndexMetadataParams {` | Ported | Keep target-branch context; preserve the source change. | | `gensrc/thrift/PlanNodes.thrift` | `@@ -860,6 +883,7 @@ struct TMetaScanRange {` | Ported | Keep target-branch context; preserve the source change. | | `gensrc/thrift/Types.thrift` | `@@ -671,6 +671,7 @@ enum TMetadataType {` | Ported | Keep target-branch context; preserve the source change. | | `regression-test/data/external_table_p0/tvf/test_index_disk_usage_tvf.out` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) | | | `regression-test/suites/external_table_p0/tvf/test_index_disk_usage_tvf.groovy` | `@@ -0,0 +1,187 @@` | Adapted | Use the established branch-4.2 config-sensitive VARIANT fixture constructor: implicit JSON string conversion for legacy VARIANT and parse_to_variant for VARIANT V2. Preserve the same 1000 JSON documents, typed path, queries, assertions and expected output. | </details> <details> <summary>Range-diff review</summary> The source squash and backport each contain one commit. The range-diff entries were reviewed as follows: - Commit metadata: retain Jack as author and the real `cherry picked from` source SHA; add the target branch, source link, and adaptation summary. - Reader: `_do_get_next_block` becomes `get_next_block`; use `get_data_type_ptr`; omit the unsupported `ReaderInitContext` override because the target scanner directly calls `init_reader`. - Collector: replace `rowset->segments()` with existing segment IDs, paths, and rowset metadata. Keep direct footer reads, row labels, query IO context, cancellation, and error propagation. The ranges calls and explicit status branches preserve ordering and skipped-container behavior. - Storage CMake: omit only the unity-skip hunk because this branch has no Storage unity batches. - Rowset test fixture: replace two calls to `create_storage_block()` with `create_block()`. Both construct the same two integer columns; no assertion is removed or weakened. - Regression fixture: match the configured VARIANT representation, using implicit JSON string conversion for legacy VARIANT and `parse_to_variant` for V2. The same 1000 documents and all assertions and expected output are retained. - Session variables: use the target annotation type and two-description shape, preserving both defaults and forwarding. - FE TVF and test: import the target `IndexDef.IndexType`, with the same enum values and assertions. - Context entries in the meta scanner, page decoder, TVF registry/visitor, session-variable tests, and Thrift enum retain the target branch's existing code. No unrelated master imports, query-cache tests, scanner defaults, or enum reservations are copied. </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
