sunchao commented on code in PR #5788:
URL: https://github.com/apache/datafusion-comet/pull/5788#discussion_r3965353004
##########
spark/src/test/scala/org/apache/spark/sql/benchmark/CometShuffleBenchmark.scala:
##########
@@ -492,6 +567,25 @@ object CometShuffleBenchmark extends CometBenchmarkBase {
}
}
+ runBenchmarkWithTable("Nested hash partitioning key", 1024 * 1024 * 1) { v
=>
+ // Shapes whose leaves are primitives take the vectorized element path;
the last two force
+ // the per-element path in the native kernel.
+ val shapes = Seq(
+ "struct<int, string>" -> "named_struct('a', c1, 'b', CAST(c1 AS
STRING))",
+ "array<int>" -> "ARRAY_REPEAT(c1, 10)",
+ "struct<array<int>, string>" ->
+ "named_struct('a', ARRAY_REPEAT(c1, 10), 'b', CAST(c1 AS STRING))",
+ "array<struct<int, string>>" ->
+ "ARRAY_REPEAT(named_struct('a', c1, 'b', CAST(c1 AS STRING)), 10)",
+ "struct<map<string, int>, int>" ->
+ "named_struct('m', MAP(CAST(c1 AS STRING), c1), 'i', c1)")
Review Comment:
### Correctness
[P2] Skip unsupported map-key native benchmark cases
Could we restrict this map-containing native case to supported Spark
versions and check that the planned exchange is `CometNativeShuffle` before
timing it? On the supported Spark 3.5 profile, `nativeShuffleFailureReasons`
recursively reaches this `MapType` and rejects it because `isSpark40Plus` is
false, even with the nested config enabled. In native-only mode the rule
retains Spark's `ShuffleExchangeExec`, but this helper still records the run as
`Comet (Native Shuffle)`. The existing map-key shuffle test explicitly expects
zero native exchanges below 4.0. These timings cannot detect native map-hashing
regressions. A version guard or explicit skip plus an untimed plan check would
keep the label accurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
