Ammar Chalifah created SPARK-60203:
--------------------------------------
Summary: SPJ with allowJoinKeysSubsetOfPartitionKeys fails in AQE
skew-join validation when a non-join partition key comes first
Key: SPARK-60203
URL: https://issues.apache.org/jira/browse/SPARK-60203
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.0.4, 4.1.3, 4.1.2, 4.1.1
Reporter: Ammar Chalifah
In copy-on-write MERGE operations with Iceberg, for tables that are partitioned
on superset of the join keys and join keys doesn't include the first partition
key, the operations fails on query planning side.
For example, a table partitioned on `date, bucket(12, id)`, and MERGE with `id`
as the only join key, raised an error:
{{```}}
{{{}java.lang.IllegalStateException: Unknown type for int field. Type name:
java.lang.String{}}},
```
The plan fails at {{{}KeyGroupedPartitioning.createShuffleSpec{}}}, then
{{{}KeyGroupedPartitioning.apply{}}}, then {{project}} over
{{{}originalPartitionValues{}}}. The scan has already been cut down to the join
keys, but {{originalPartitionValues}} were never projected, so Spark reads the
full (date, id) row with the join-key types.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
