viirya opened a new issue, #5787:
URL: https://github.com/apache/datafusion-comet/issues/5787
### What is the problem the feature request solves?
`spark.comet.shuffle.native.partitioning.hash.nested.enabled` (#5567) lets a
struct, array or
map be a native shuffle hash partitioning key. Nothing in
`CometShuffleBenchmark` measures those
shapes, so there is no baseline for whether native hashing of a nested key
beats letting Spark do
the shuffle, and no way to see a regression in that path.
The existing cases do not cover it:
- `shuffleArrayBenchmark` and `shuffleStructBenchmark` repartition on a
nested column but have no
native-shuffle case, because the gate rejected nested keys when they were
written.
- `shuffleDeeplyNestedBenchmark` calls `repartition(n)` with no key, which
is round robin rather
than hash partitioning, so it never exercises the hash kernel.
This is separate from #2904, which was about columnar/JVM shuffle throughput
on nested schemas
with a keyless `repartition(n)`.
### Describe the potential solution
Add a `shuffleNestedHashKeyBenchmark` with Spark, Comet (Spark Shuffle),
Comet (JVM Shuffle) and
Comet (Native Shuffle) cases, where the native case enables the nested
config so it measures the
native hashing path rather than a silent fallback to Spark's shuffle.
Cover shapes that separate the two code paths in the native Murmur3 kernel:
a list whose elements
are primitives is vectorized, while a list whose elements are themselves
nested falls through to
`hash_list_array!`. So `struct<int, string>`, `array<int>`,
`struct<array<int>, string>`,
`array<struct<int, string>>` and `struct<map<string, int>, int>`, at a small
and a large partition
count.
### Additional context
Measuring this is what surfaced #5777: `array<struct<..>>` as a native hash
key runs at 0.3-0.4x
of leaving the shuffle to Spark, while every other shape measured is faster
natively. The
benchmark should land independently of any kernel change so the before/after
comparison has a
reference point that is not moving at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
