Doris-Breakwater commented on issue #69782: URL: https://github.com/apache/doris/issues/69782#issuecomment-6473294107
Breakwater-GitHub-Analysis-Slot: slot_e6d6cdcbb651 **Assessment:** the expression lifecycle defect is confirmed by source inspection at both reported revisions (`9f63be85cea` / branch-4.2 and `5fc51d21256` / branch-4.1). The specific BE abort on those branches remains unverified: the missing piece is a reachable 4.x plan that puts the cast under this expression. Treat this as an actionable Iceberg/FileScannerV2 lifecycle issue with a potentially serious crash consequence, while retaining that qualification. **Verified mechanism** - `prepare()` sets `_prepared`. The guard in `open()` then returns before evaluating `VExpr::open(...)`, so neither the children nor the assignment to `_open_finished` is reached. This is present in [4.2 `AncestorNullDefaultExpr`](https://github.com/apache/doris/blob/9f63be85cead9fb8f295dd6ac74ebf0c6f1bfa8e/be/src/format_v2/table/iceberg_reader.cpp#L722-L741) and [4.1](https://github.com/apache/doris/blob/5fc51d21256d598dc89afe5732550b62baeac4a9/be/src/format_v2/table/iceberg_reader.cpp#L722-L741); see the [guard definition](https://github.com/apache/doris/blob/9f63be85cead9fb8f295dd6ac74ebf0c6f1bfa8e/be/src/exprs/vexpr.h#L75-L81). The surrounding `EqualityDeletePredicate` correctly opens its children, but cannot compensate for this child returning early. - Given the reported type mismatch, the missing-key builder does attach `EvolutionCast` on 4.2, or `Cast` on 4.1, below `AncestorNullDefaultExpr`. `EvolutionCast` inherits `Cast::open()`, which initializes the function context. Execution checks `_open_finished || block == nullptr` with **`DCHECK`**, so the exact assertion abort depends on the build's DCHECK configuration; an unopened function context remains a lifecycle violation regardless. See the [builder](https://github.com/apache/doris/blob/9f63be85cead9fb8f295dd6ac74ebf0c6f1bfa8e/be/src/format_v2/table/iceberg_reader.cpp#L1419-L1472) and [Cast lifecycle](https://github.com/apache/doris/blob/9f63be85cead9fb8f295dd6ac74ebf0c6f1bfa8e/be/src/format_v2/expr/cast.cpp#L72-L133). - The default child is evaluated **before** the per-row ancestor NULL mask is applied. An all-NULL parent batch therefore does not avoid execution of an existing cast child. The key result should still be NULL under a NULL ancestor; this is distinct from short-circuiting evaluation. See [execution order](https://github.com/apache/doris/blob/9f63be85cead9fb8f295dd6ac74ebf0c6f1bfa8e/be/src/format_v2/table/iceberg_reader.cpp#L747-L789). **What is not yet established on 4.x** A table/delete-file type difference alone does not prove the required cast is constructed. In these revisions, [`_resolve_equality_delete_fields`](https://github.com/apache/doris/blob/9f63be85cead9fb8f295dd6ac74ebf0c6f1bfa8e/be/src/format_v2/table/iceberg_reader.cpp#L1694-L1725) already restores the table type into `filter.key_types`. The later table-path lookup also restores primitive types from the external schema through [`restore_current_primitive_type`](https://github.com/apache/doris/blob/9f63be85cead9fb8f295dd6ac74ebf0c6f1bfa8e/be/src/format_v2/table_reader.cpp#L527-L555). With consistent metadata, these two types ordinarily agree. Please provide the concrete differing types and schema carriers that make condition 3 true on each branch; do not assume that the new per-file comparison behavior in #69711 already exists in 4.x. I inspected the named test in [PR commit `3c2a8e9cb87`](https://github.com/apache/doris/blob/3c2a8e9cb87231251e18d7f397754222db69b75e/be/test/format_v2/table/iceberg_reader_test.cpp#L5565-L5640): it supplies a STRING schema placeholder and text default, then exercises the PR's new tombstone-type comparison path. That is useful regression coverage for the lifecycle fix, but is not itself a reproduction against unchanged 4.x. Also, [#69711](https://github.com/apache/doris/pull/69711) is currently **open and unmerged**; the cited commit is its head, not evidence of a released fix. Its current backport label is `backport/4.2-master`. **Separate TIME / timestamptz_ns concern** The 4.2 source supports the reported transport hazard: [Iceberg type mapping](https://github.com/apache/doris/blob/9f63be85cead9fb8f295dd6ac74ebf0c6f1bfa8e/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java#L817-L828) returns UNSUPPORTED for TIME and zoned nanosecond timestamps; the [schema carrier](https://github.com/apache/doris/blob/9f63be85cead9fb8f295dd6ac74ebf0c6f1bfa8e/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalUtil.java#L227-L236) serializes the column type, and BE type restoration calls `thrift_to_type`, whose [default is an unconditional `CHECK(false)`](https://github.com/apache/doris/blob/9f63be85cead9fb8f295dd6ac74ebf0c6f1bfa8e/be/src/core/data_type/primitive_type.cpp#L150-L153). If that carrier reaches equality-key resolution, it can fail before default evaluation. This is a separate failure path; an end-to-end FE/BE reproduction is still needed to establish whether planning rejects the input earlier. Opening expression ch ildren alone does not resolve it. **Recommended next steps / missing evidence** 1. Backport the small lifecycle correction from #69711 for review on each affected branch: use the normal prepare-finished precondition and call `VExpr::open` without the prepare-state guard. Keep it accompanied by a focused test proving that a cast child is actually opened. Audit `NestedStructFieldExpr::open()` too: it has the same guard pattern, although no separate crash is established here. 2. Add a branch-native reproduction using an old data file with `s` present and `s.k` absent, a non-NULL initial default, and a concrete equality-delete key type that reaches the cast branch. Record `table_leaf->type`, `filter.key_types[key_idx]`, and the constructed expression tree. Cover NULL/non-NULL parents, matching/nonmatching/NULL tombstones, and Parquet/ORC as supported. Check row IDs as well as survival of the BE. 3. For the second concern, test a query projecting only a supported column while a hidden equality-delete key is TIME or `timestamptz_ns`; inspect the query-wide and split-specific serialized schema types. Handle unsupported input with defined semantics or a controlled error, and verify any placeholder change together with default parsing and comparison behavior. 4. Attach the minimal Iceberg metadata/schema history (field IDs, initial defaults, selected snapshot), data/delete file schemas or sanitized fixtures, producing engine/version, SQL, FileScannerV2 settings, exact FE/BE build hashes and DCHECK/build mode. If a 4.x crash is observed, include the fatal log and symbolized stack around `Cast` or `thrift_to_type`, plus query ID. These distinguish the two failure paths. Validation scope: read-only inspection of the exact branch snapshots and PR head, plus live issue/PR metadata. No Doris build or unit test was run, no 4.x crash was reproduced, and no code was changed. The issue currently has no labels; suggested triage is bug / Iceberg / FileScannerV2 with branch-4.1 and branch-4.2 tracking, prioritizing a confirmed reachable BE abort once reproduced. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
