924060929 opened a new pull request, #70666: URL: https://github.com/apache/doris/pull/70666
### Background Nereids stores equivalent plans in Memo groups. Regular logical and physical expressions connect these groups into a directed acyclic graph, and statistics derivation visits children before their parents. When an inserted expression already exists in another group, `Memo.mergeGroup` combines the groups and redirects their parent expressions. ### What problem does this PR solve? Issue Number: close #70174 Related PR: None PR Type: Bugfix Problem Summary: A one-key join of two tables with correlated `EXISTS` row policies can exhaust the FE heap during planning with sampled statistics. Project merging finds an existing expression in an indirect descendant group, but the old merge guard checks only immediate parent-child relationships. This change rejects a merge whenever either group can reach the other through regular logical or physical expressions. Most of the diff adds unit and SQL regression coverage; valid merges with shared children and enforcers remain supported. ### Details - **Change map:** 39 changed lines in `Memo.java`; 108 unit-test lines, 180 SQL-regression lines, and 26 automatically generated result lines. - **Frequency:** Deterministic with the sampled-statistics, single-key reproducer in #70174 on 4.1.5, for both plain `EXISTS` and `OR EXISTS` policies. No production incident-frequency estimate. - **Root cause:** `ApplyRuleJob -> Memo.copyIn -> insertGroupExpression -> mergeGroup` contracts an ancestor and an indirect descendant. A reduced example is `G3(Project k) -> G2(Project 1,k) -> G1(Project k) -> G0(Scan k)`: inserting `Project(k,G0)` into G3 finds G1, and merging G3 into G1 creates `G1 -> G2 -> G1`. `DeriveStatsJob` then keeps scheduling unfinished children; a reproduced 2 GiB OOM heap contained about 41.9 million `DeriveStatsJob` objects, with only 2,561 Memo expressions. - **Why existing tests missed it:** Existing merge tests cover ordinary merges and direct relationships, but do not validate the indirect cycle produced when project merging rediscovers an expression in a descendant group. - **Fix:** Check reachability in both directions before mutating either group. Use an iterative traversal with a visited set to handle shared descendants without recursive stack growth. Enforcers are stored separately and intentionally excluded because their self-reference is valid. - **Other paths and branches:** The common entry also protects recursive parent-expression deduplication. Tests cover direct and deeper logical dependencies, an indirect physical dependency, and a permitted merge with shared children and an enforcer. This PR targets master; the same patch was independently built and exercised on the 4.1.5 source base. The separately reported cloud warmup-timeout behavior is outside this group-cycle fix. - **Behavior change:** Merges that would create an indirect dependency cycle are refused, as direct-cycle merges already were. No configuration, RPC, or persisted format changes. Reachability checks are linear in the reachable groups and expression edges; the SQL timings below are functional validation, not a planner-throughput benchmark. ### Testing - Before the fix, the reduced `testIndirectProjectMergeMustNotCreateCycle` fails `MemoValidator` on the 4.1.5 base. With the patch, `MemoTest` runs 42 tests with 0 failures, 0 errors, and 1 existing skipped test. - Master: `MemoTest`, `GroupExpressionTest`, and `StructInfoMapTest` through `run-fe-ut.sh`: 47 tests, 0 failures, 0 errors, 1 existing skipped test. The final test initialization uses the existing `MemoTestUtils` factory to supply master's statement context. - Clean master FE build with Checkstyle: `MVN_OPT='-Dmaven.build.cache.enabled=false' DISABLE_BUILD_UI=ON DISABLE_BE_JAVA_EXTENSIONS=ON ./build.sh --fe --clean -j4 --output output/fixed-master`. - Patched 4.1.5 FE with original 4.1.5 BE, ARM64, 2 GiB FE heap: the unmodified public issue's single-key `EXPLAIN`, sampled statistics, and no disabled rules complete in 0.591 s (plain policy) and 0.297 s (OR policy). Both previously exhausted the same heap. - The 4.1.5 SQL regression output was generated by `run-regression-test.sh`, inspected, then checked by a second run. It covers both policy forms, one/two keys, admin bypass, a granted reader, and a rejected reader; granted/admin cases have nonempty expected rows. - Master FE with original 4.1.5 BE, ARM64, 2 GiB FE heap: after rerunning sampled analysis, the public single-key EXPLAIN completes in 0.081 s (plain policy) and 0.167 s (OR policy), with no disabled rules and FE survival checked. The full master SQL suite could not execute against this older BE: its first fixture INSERT is rejected with `Unsupported exec type in pipeline: Invalid plan node type`. Full matching-version master FE/BE regression remains for CI; this is not counted as a passing master runtime suite. ### Release note Fix FE out-of-memory failures during planning when equivalent Memo groups have an indirect dependency, including joins protected by correlated EXISTS row policies. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: local master unit tests, FE build and Checkstyle, and the patched 4.1.5 end-to-end regression passed. Full matching-version master FE/BE SQL regression is pending CI (see Testing). ### AI Disclosure - Tool: OpenAI Codex. - Generated: Production fix, unit tests, SQL regression, and this description. The result file was generated by the regression runner. - Human review: Pending author review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
