Peter Toth created SPARK-59120:
----------------------------------
Summary: Fix ClassCastException in a storage-partitioned join
whose partition keys were reduced
Key: SPARK-59120
URL: https://issues.apache.org/jira/browse/SPARK-59120
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
With {{spark.sql.sources.v2.bucketing.allowCompatibleTransforms.enabled}} a
storage-partitioned join reduces one or both sides' partition keys onto a
common key space, while the partitioning keeps reporting the partition
expressions it was built from. The stored key rows are then no longer what
those expressions describe, and where the reduction also changes the type,
reading a key row at the expression's type unboxes an Integer as a Long. Two
queries fail with {{ClassCastException}} because of that.
*1. At planning.* Two tables partitioned by {{identity(ts)}} and by
{{years(ts)}}, joined on {{ts}}, with the subset opt-in on. AQE re-runs
{{createShuffleSpec}} on the already reduced children through
{{ValidateRequirements}}, which projects and sorts the keys, and the sort
orders {{IntegerType}} year values with a comparator built for
{{TimestampType}}:
{noformat}
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class
java.lang.Long
at KeyedPartitioning.toGrouped
at KeyedPartitioning.createShuffleSpec
at ValidateRequirements
{noformat}
*2. While the shuffle is prepared.* The same two tables plus a third,
unpartitioned one joined on the same column, with
{{spark.sql.sources.v2.bucketing.shuffle.enabled}} on. A reduced partitioning
is grouped and does satisfy the distribution, so {{EnsureRequirements}} offers
it as the layout to shuffle the third child onto. {{ShuffleExchangeExec}} then
builds the shuffle's key map by re-wrapping the stored keys at the expressions'
types, and the same unboxing throws on the driver.
The reduce, and the reporting of reduced keys under the original expressions,
arrived in SPARK-47094 (4.0.0). The crashes start in 4.2.0, with the
{{KeyedPartitioning}} / {{GroupPartitionsExec}} refactor that derives the key
ordering and types from the reported expressions. On 4.1 and earlier the same
key-space mismatch is there, but the symptom is wrong results rather than an
exception, and the code is different enough to need its own ticket.
Not in scope here: a reduction that keeps the type, {{bucket(12)}} and
{{bucket(8)}} both reducing onto {{bucket(4)}}, still misroutes rows silently.
Answering that needs the partitioning to say whether its expressions still
describe its keys, which is a follow-up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
