ahmed-mez opened a new pull request, #16506: URL: https://github.com/apache/datafusion/pull/16506
## Which issue does this PR close? Fixes stack overflows caused by deeply nested query plans during recursive function calls in various optimizer and expression evaluation paths. ## Rationale for this change DataFusion was experiencing stack overflow panics when processing deeply nested query plans. The issue occurred because many functions throughout the codebase perform recursive operations on query plans and expressions without stack overflow protection. When these operations encounter deeply nested structures, they can exhaust the call stack, causing the application to crash. This change extends the existing recursive protection mechanism to additional functions across the optimizer, physical expression planner, and other core components that were previously unprotected. Note that https://github.com/apache/datafusion/pull/16404 also helped mitigate the stack overflows I encountered with my reproducer, very likely because cloning `Expr` is not stack friendly especially when `Expr` is deeply nested. This is another data point in favour of https://github.com/apache/datafusion/issues/9577. ## What changes are included in this PR? - Added `#[cfg_attr(feature = "recursive_protection", recursive::recursive)]` annotations to various recursive functions across multiple crates - Extended the `recursive_protection` feature to `datafusion-physical-expr` and `datafusion-substrait` crates ## Are these changes tested? The changes are primarily defensive additions that activate existing recursion protection when the `recursive_protection` feature is enabled. The recursion protection mechanism itself is already used in the codebase. Ideally we should work on automating the detection of lacking recursion protection. About the reproducer: it's a 5MB+ Substrait JSON plan that's deeply nested (but valid). I don't see value in sharing it publicly as it contains various custom extensions and can't be used without an adequate Datadog-specific setup. However It helped me debug and validate the proposed changes. ## Are there any user-facing changes? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
