Simon Maling created SPARK-59574:
------------------------------------
Summary: Single-pass Analyzer does not apply
injectHintResolutionRule
Key: SPARK-59574
URL: https://issues.apache.org/jira/browse/SPARK-59574
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.0, 4.2.0, 4.3.0, 4.0.0
Reporter: Simon Maling
The single-pass Resolver never applies the analyzer hint-resolution rules
contributed by
SparkSessionExtensions.injectHintResolutionRule.
The fixed-point Analyzer runs them in the "Hints" batch of earlyBatches, ahead
of the
"Resolution" batch that contains ResolveSQLOnFile:
{code:scala}
Batch("Hints", fixedPoint,
Seq(ResolveHints.ResolveJoinStrategyHints,
ResolveHints.ResolveCoalesceHints) ++
hintResolutionRules: _*),
{code}
Both session builders wire the extension rules into that list via
{{hintResolutionRules = customHintResolutionRules}} (HiveSessionStateBuilder
since SPARK-59081).
The single-pass Resolver has no equivalent. HybridAnalyzer.fromLegacyAnalyzer
forwards
singlePassResolverExtensions, singlePassMetadataResolverExtensions,
singlePassPostHocResolutionRules and singlePassExtendedResolutionChecks, but not
hintResolutionRules, so those rules are silently dropped when:
* spark.sql.analyzer.singlePassResolver.enabled=true
* spark.sql.analyzer.singlePassResolver.enabledTentatively=true
* in the single-pass half of
spark.sql.analyzer.singlePassResolver.dualRunWithLegacy
h3. Impact
1. Rules registered through a documented public API do not run under the
single-pass Analyzer.
2. Rules that must run *before* relation metadata lookup cannot work at all.
Example: path SQL
such as {{SELECT * FROM parquet.`s3://bucket/dir`}}. Under fixed-point, a hint
rule sees the
UnresolvedRelation and can rewrite its options before ResolveSQLOnFile. Under
single-pass,
FileResolver (a singlePassMetadataResolverExtensions entry that reuses
ResolveSQLOnFile)
resolves and lists the path in the metadata pre-pass, before any extension code
can touch
the relation.
3. Dual-run compares the fixed-point plan against the single-pass plan, so any
extension using
injectHintResolutionRule can make the two diverge.
h3. Why existing hooks are not a substitute
* injectPostHocResolutionRule reaches singlePassCustomPostHocResolutionRules,
which runs after
resolution, i.e. after the metadata pre-pass has already listed files.
* singlePassMetadataResolverExtensions is constructed in
BaseSessionStateBuilder /
HiveSessionStateBuilder and is not injectable through SparkSessionExtensions.
This is the same class of gap as SPARK-59081 and SPARK-56453: an analyzer rule
list that was not
updated when a second analyzer path was introduced.
h3. Proposal
Pass Analyzer.hintResolutionRules into Resolver and apply them in
lookupMetadataAndResolve after
IDENTIFIER/CTE substitution and before the metadata lookup, mirroring the
fixed-point ordering
(Substitution -> Hints -> ... -> Resolution/ResolveSQLOnFile). No behavior
change when no
extension registers hint rules.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
