andygrove opened a new issue, #6593:
URL: https://github.com/apache/datafusion-comet/issues/6593
### What is the problem the feature request solves?
`spark.comet.sparkToColumnar.enabled` and
`spark.comet.sparkToColumnar.supportedOperatorList` decide which Spark leaf
operators Comet converts to Arrow, so that the operators above them can run
natively. The list is a poor interface for this:
- Its entries are Spark physical class names without the `Exec` suffix, and
those change between Spark versions. Spark 4.1 added `OneRowRelationExec`. On
Spark 3.4 through 4.0, Spark plans a `OneRowRelation` as an `RDDScanExec` named
`OneRowRelation`. So the default `OneRowRelation` entry matches nothing there,
and removing `RDDScan` from the list also stops converting queries such as
`SELECT 1`.
- `FileSourceScan` and `BatchScan` only reach formats other than Parquet,
JSON and CSV. Those three check
`spark.comet.convert.{parquet,json,csv}.enabled` first.
- The default list has no effect unless
`spark.comet.sparkToColumnar.enabled` is set, and that switch defaults to false.
- Turning on one case takes two settings. Setting the list replaces the
default rather than adding to it, and names aren't validated, so a typo or a
name with the `Exec` suffix silently does nothing.
- One switch covers several cases with different trade-offs, so no case can
have its own default or its own documentation. #832 added the per-format
`spark.comet.convert.*` configs for the same reason, and #6564 adds
`spark.comet.convert.typedDataset.enabled`.
### Describe the potential solution
Give each entry in the default list its own config, off by default, next to
the per-format ones:
| List entry | New config
|
| ------------------- |
---------------------------------------------------------------------------------------------------------
|
| `Range` | `spark.comet.convert.range.enabled`, for ranges that
`CometRangeExec` does not generate |
| `InMemoryTableScan` | `spark.comet.convert.inMemoryCache.enabled`, for
cached tables that the native cache scan doesn't read |
| `RDDScan` | `spark.comet.convert.rdd.enabled`, for DataFrames
created from an RDD of rows |
| `OneRowRelation` | `spark.comet.convert.oneRowRelation.enabled`, for
queries without a `FROM` clause, on every Spark version |
Keep the list for leaf operators that have no config of their own: Data
Source V2 connectors (all of them are a `BatchScanExec`), file formats such as
ORC and text, JDBC, and Hive tables. Its default becomes empty.
Deprecate the use of `spark.comet.sparkToColumnar.enabled` and the list for
the four cases above. Until that is removed, they keep working as they do
today, with a one-time warning that names the new config:
- Turning on the switch without setting the list still converts all four.
- A list that names one of them still converts it.
No default changes, so behavior stays the same until a user sets one of the
new configs.
### Additional context
Possible follow-ups, out of scope here:
- Turning `spark.comet.convert.oneRowRelation.enabled` on by default. The
conversion is a single row with no columns, and it may help #4949.
- Per-format configs for ORC and text (#6008).
- Folding the `Range` conversion into `spark.comet.exec.range.enabled`, as
the fallback when `CometRangeExec` declines a range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
