andygrove opened a new issue, #6891:
URL: https://github.com/apache/datafusion-comet/issues/6891
### What is the problem the feature request solves?
#6776 adds a contributor guide rule that protobuf fields, native arguments,
and native function names that vary with the Spark version describe the
behavior rather than the version. Two existing names predate it.
`Cast.is_spark4_plus` in `expr.proto` is set from `isSpark40Plus` in
`CometCast.scala`. Its comment says it controls version-specific cast behavior
"such as" leading whitespace before T-prefixed time-only strings, which invites
putting more behavior behind it. Today it controls only that one case. Since
SPARK-46488 (Spark 4.0), Spark recognizes the `T` of a time-only string only at
the first byte, so `CAST(' T2' AS TIMESTAMP)` is null, or an error under ANSI,
where Spark 3.x trims first and accepts it. `timestamp_ntz_parser` takes the
flag and ignores it.
`get_json_object_spark34` is the native variant that skips Jackson's
1000-digit number limit. Since #6600 the Spark 3.4 shim picks it only when the
Jackson on the classpath is older than 2.15, so Spark 3.4 with a newer Jackson
runs the plain `get_json_object`. The name no longer says when the variant runs
or what it does.
### Describe the potential solution
Rename `is_spark4_plus` for the behavior, for example
`reject_padding_before_time_only_t`, along with the Rust side
(`SparkCastOptions` and the parser arguments). Protobuf encodes the field
number rather than the name, so the rename is wire-compatible, and the JVM and
native sides ship together anyway. A later Spark change to another cast
behavior should get its own field rather than a second branch on this one.
Rename `get_json_object_spark34` for what it does, for example
`get_json_object_unlimited_number_length`, or register a single function that
takes the limit as an argument.
### Additional context
Raised in the review of #6776.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
