comphead commented on code in PR #6698:
URL: https://github.com/apache/datafusion-comet/pull/6698#discussion_r4197476163
##########
docs/source/user-guide/latest/tuning.md:
##########
@@ -36,6 +36,20 @@ guide is split into the following pages:
- [Reducing Row/Columnar Conversion Overhead](tuning/transitions.md): stages
in which many
operators fall back to Spark.
+## Nested and Wide Data
+
+Rows with many columns, long strings, or nested columns such as arrays of
structs make every batch
+larger, so queries over such data spill more. For these workloads:
+
+- Compress spill files, which Comet writes uncompressed by default. See
Review Comment:
Reworded. The bullet now names native sorts, aggregations, and joins, and
reflects the new `lz4` default.
##########
spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala:
##########
@@ -110,6 +110,21 @@ class CometExecSuite extends CometTestBase {
}
}
+ test("SQLConf serde passes allowed DataFusion configs without
respectDataFusionConfigs") {
+ def entries =
ConfigMap.parseFrom(CometExecIterator.serializeCometSQLConfs()).getEntriesMap
+ val spillCompression = "spark.comet.datafusion.execution.spill_compression"
+ val spillReservation =
"spark.comet.datafusion.execution.sort_spill_reservation_bytes"
+
+ withSQLConf(spillCompression -> "zstd", spillReservation -> "65536") {
+ assert(entries.get(spillCompression) == "zstd")
+ assert(!entries.containsKey(spillReservation))
+
+ withSQLConf(CometConf.COMET_ALLOWED_DATAFUSION_CONFIGS.key ->
spillReservation) {
Review Comment:
The allowlist is gone, so this test was replaced by one that compares the
`spilled_bytes` of a spilling native sort with the default codec and with
`none`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
