[
https://issues.apache.org/jira/browse/SPARK-59574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59574:
-----------------------------------
Labels: pull-request-available (was: )
> Single-pass Analyzer does not apply injectHintResolutionRule
> ------------------------------------------------------------
>
> Key: SPARK-59574
> URL: https://issues.apache.org/jira/browse/SPARK-59574
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.0, 4.0.0, 4.2.0, 4.3.0
> Reporter: Simon Maling
> Priority: Major
> Labels: pull-request-available
>
> The single-pass Resolver never applies the analyzer hint-resolution rules
> contributed by
> SparkSessionExtensions.injectHintResolutionRule.
> The fixed-point Analyzer runs them in the "Hints" batch of earlyBatches,
> ahead of the
> "Resolution" batch that contains ResolveSQLOnFile:
> {code:scala}
> Batch("Hints", fixedPoint,
> Seq(ResolveHints.ResolveJoinStrategyHints,
> ResolveHints.ResolveCoalesceHints) ++
> hintResolutionRules: _*),
> {code}
> Both session builders wire the extension rules into that list via
> {{hintResolutionRules = customHintResolutionRules}} (HiveSessionStateBuilder
> since SPARK-59081).
> The single-pass Resolver has no equivalent. HybridAnalyzer.fromLegacyAnalyzer
> forwards
> singlePassResolverExtensions, singlePassMetadataResolverExtensions,
> singlePassPostHocResolutionRules and singlePassExtendedResolutionChecks, but
> not
> hintResolutionRules, so those rules are silently dropped when:
> * spark.sql.analyzer.singlePassResolver.enabled=true
> * spark.sql.analyzer.singlePassResolver.enabledTentatively=true
> * in the single-pass half of
> spark.sql.analyzer.singlePassResolver.dualRunWithLegacy
> h3. Impact
> 1. Rules registered through a documented public API do not run under the
> single-pass Analyzer.
> 2. Rules that must run *before* relation metadata lookup cannot work at all.
> Example: path SQL
> such as {{SELECT * FROM parquet.`s3://bucket/dir`}}. Under fixed-point, a
> hint rule sees the
> UnresolvedRelation and can rewrite its options before ResolveSQLOnFile. Under
> single-pass,
> FileResolver (a singlePassMetadataResolverExtensions entry that reuses
> ResolveSQLOnFile)
> resolves and lists the path in the metadata pre-pass, before any extension
> code can touch
> the relation.
> 3. Dual-run compares the fixed-point plan against the single-pass plan, so
> any extension using
> injectHintResolutionRule can make the two diverge.
> h3. Why existing hooks are not a substitute
> * injectPostHocResolutionRule reaches singlePassCustomPostHocResolutionRules,
> which runs after
> resolution, i.e. after the metadata pre-pass has already listed files.
> * singlePassMetadataResolverExtensions is constructed in
> BaseSessionStateBuilder /
> HiveSessionStateBuilder and is not injectable through SparkSessionExtensions.
> This is the same class of gap as SPARK-59081 and SPARK-56453: an analyzer
> rule list that was not
> updated when a second analyzer path was introduced.
> h3. Proposal
> Pass Analyzer.hintResolutionRules into Resolver and apply them in
> lookupMetadataAndResolve after
> IDENTIFIER/CTE substitution and before the metadata lookup, mirroring the
> fixed-point ordering
> (Substitution -> Hints -> ... -> Resolution/ResolveSQLOnFile). No behavior
> change when no
> extension registers hint rules.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
