andygrove opened a new pull request, #6349:
URL: https://github.com/apache/datafusion-comet/pull/6349
## Which issue does this PR close?
Closes #6332.
## Rationale for this change
The native CSV V2 scan hands DataFusion's `CsvSource` the Spark schema,
where `TimestampType` is `Timestamp(Microsecond, "UTC")`. As a result,
arrow-csv reads a timestamp without an offset as UTC.
Spark's CSV reader interprets that same value in the CSV `timeZone` option,
which defaults to the session timezone. Nothing passes that timezone to the
native reader, because the `CsvOptions` proto has no field for it. In a non-UTC
session, every such value was therefore silently shifted. For example, in
`America/Los_Angeles`, `2024-01-15 18:30:45` came back as
`2024-01-15T18:30:45Z` instead of `2024-01-16T02:30:45Z`.
## What changes are included in this PR?
- The CSV V2 branch of `CometScanRule` now falls back to Spark when the read
schema has a `TimestampType` column and the CSV timezone isn't UTC. The CSV
timezone is the `timeZone` option, or the session timezone when that option
isn't set.
- `TimestampNTZType` is unaffected, because neither reader applies a
timezone to it.
- The check normalizes the zone, so `UTC`, `Etc/UTC`, `Z` and `+00:00` all
stay native.
- A new test in `CometCsvNativeReadSuite` covers four cases:
- a Los Angeles session falls back
- a Los Angeles session with `timeZone=UTC` stays native
- a UTC session stays native
- a UTC session with `timeZone=Asia/Tokyo` falls back
The scan is testing-only and off by default. Passing the timezone to the
native reader would need a proto field and a native change to how timestamps
are parsed, so this PR just makes the unsupported case fall back.
## How are these changes tested?
- With the old rule, the new test fails because the results don't match in
the Los Angeles session.
- With this change, all four `CometCsvNativeReadSuite` tests pass on Spark
4.1, including the `spotless` and `scalastyle` checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
