[
https://issues.apache.org/jira/browse/SPARK-57758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-57758.
------------------------------
Fix Version/s: 4.2.1
4.3.0
Resolution: Fixed
Issue resolved by pull request 56869
[https://github.com/apache/spark/pull/56869]
> Built-in function resolution is no longer O(1) after SPARK-54807, regressing
> Spark Connect AnalyzePlan
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-57758
> URL: https://issues.apache.org/jira/browse/SPARK-57758
> Project: Spark
> Issue Type: Bug
> Components: Connect, SQL
> Affects Versions: 4.2.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.1, 4.3.0
>
>
> SPARK-54807 (#53570) added qualified function names and a configurable
> resolution search path ({{spark.sql.functionResolution.sessionOrder}}). As a
> side effect it changed how
> *unqualified* function names are resolved, introducing a performance
> regression in the analyzer.
> Previously, an unqualified built-in function (e.g. {{count}}, {{coalesce}},
> {{sum}}) resolved with a single in-memory registry lookup. After SPARK-54807,
> {{FunctionResolution.resolveFunction}} (and {{resolveTableFunction}}) build
> an ordered candidate search path for *every* {{UnresolvedFunction}}. For each
> function node, per call,
> it now:
> * reads the {{AnalysisContext}} thread-local and {{CatalogManager}}
> ({{currentCatalogPath}}),
> * reads the {{spark.sql.functionResolution.sessionOrder}} conf and
> allocates the search-path {{Seq}}s ({{resolutionSearchPath}}),
> * allocates the candidate list ({{searchPath.map(_ ++ nameParts)}}), and
> * iterates candidates, each doing a name-kind parse plus a registry lookup.
> None of this is memoized across an analysis pass, so it is recomputed for
> every function node.
> For plans with many built-in function references this adds substantial
> per-function overhead. The impact is amplified under Spark Connect, which
> re-analyzes the entire (growing)
> plan on every {{AnalyzePlan}} call: the per-function overhead is paid
> repeatedly, scaling roughly with plan size x number of analyze calls, and
> produces a multi-fold regression
> in analysis time. Execution time is unaffected -- the regression is
> isolated to the analysis phase.
> *Reproduction:* a Spark Connect session that incrementally builds a wide
> plan containing many built-in function calls, comparing {{AnalyzePlan}}
> latency against a pre-SPARK-54807
> build.
> h3. Proposed fix
> # *Built-in fast-path for single-part names.* In
> {{resolveFunction}}/{{resolveTableFunction}}, before constructing the
> candidate search path, resolve a single-part name directly
> against the in-memory built-in/temp registry
> ({{v1SessionCatalog.resolveBuiltinOrTempFunction}}) and return immediately on
> a hit. {{system.builtin}} is the first candidate in
> *all* {{sessionOrder}} modes ({{first}}/{{second}}/{{last}}), so a
> *built-in-only* fast-path cannot change resolution precedence.
> # *Do not fast-path session/temporary functions.* Under
> {{sessionOrder=last}} a persistent function must shadow a session function,
> so session/temp resolution must remain in the
> ordered search path. Only built-ins are safe to short-circuit.
> # *Memoize per pass.* Cache {{currentCatalogPath}} and the computed
> {{resolutionSearchPath}} for the duration of one resolution pass (they do not
> change mid-pass), eliminating
> the repeated thread-local reads and {{Seq}} allocations even for the
> temp/persistent fall-through.
> # *Minor correctness check (optional).*
> {{resolveQualifiedFunction}}/{{resolveQualifiedTableFunction}} catch
> {{AnalysisException}} with condition {{FORBIDDEN_OPERATION}} and
> return {{None}}, which can surface a genuine permission error as an
> "unresolved routine" error. Worth confirming this is intended.
> This restores the previous fast resolution for built-in-heavy plans (the
> dominant case) while preserving the full qualified-name and
> configurable-order semantics SPARK-54807
> introduced.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
