[
https://issues.apache.org/jira/browse/SPARK-59721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated SPARK-59721:
-----------------------------------------
Summary: Fall back to unknown partitioning instead of failing when a
reported partition key cannot be resolved (was: Keep storage-partitioned join
when a partition key cannot be resolved)
> Fall back to unknown partitioning instead of failing when a reported
> partition key cannot be resolved
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-59721
> URL: https://issues.apache.org/jira/browse/SPARK-59721
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Anurag Mantripragada
> Priority: Major
> Labels: pull-request-available
>
> V2ExpressionUtils.toCatalystOpt and toCatalystTransformOpt should gracefully
> return None when a data source's reported partition/bucket/transform key
> cannot be resolved against the relation. This is similar to
> V2ScanPartitioningAndOrdering's partitioning() rule (SPARK-59248) relies on
> exactly this to degrade a KeyGroupedPartitioning to unknown partitioning
> instead of failing the query when a key can't be resolved.
> Found while reviewing SPARK-58111 (PR
> [#55518|https://github.com/apache/spark/pull/55518]); this bug is general and
> pre-existing, not
> introduced there.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
