andygrove opened a new issue, #354:
URL: https://github.com/apache/datafusion-spark/issues/354

   ### Describe the bug
   
   `to_utc_timestamp` uses the wrong offset for some times on the day that 
daylight saving time starts, and it rejects time zone IDs such as `GMT+1` that 
Spark accepts.
   
   `adjust_to_utc_time` in `datetime/to_utc_timestamp.rs` treats the local time 
as if it were a UTC instant and looks up the zone's offset at that instant 
(`offset_from_utc_datetime`). On the day the clocks change, that instant can 
fall on the other side of the change from the local time. It also parses the 
zone with chrono's `Tz` rather than `TimeZoneRules`, which `from_utc_timestamp` 
uses.
   
   ### To Reproduce
   
   ```sql
   SELECT to_utc_timestamp('2024-03-10 05:00:00', 'America/Los_Angeles');
   SELECT to_utc_timestamp('2024-01-15 12:00:00', 'GMT+1');
   ```
   
   | Query | Spark 4.2.0 | datafusion-spark |
   |---|---|---|
   | `to_utc_timestamp('2024-03-10 05:00:00', 'America/Los_Angeles')` | 
`2024-03-10 12:00:00` | `2024-03-10T13:00:00` |
   | `to_utc_timestamp('2024-01-15 12:00:00', 'GMT+1')` | `2024-01-15 11:00:00` 
| `Execution error: to_utc_timestamp: invalid timezone 'GMT+1'` |
   
   `05:00` on 2024-03-10 is PDT (UTC-7), but `05:00` UTC is before the change 
at `10:00` UTC, so the PST offset (UTC-8) is used. Local times from `03:00` to 
`09:59` that day are off by an hour. For comparison, 
`from_utc_timestamp('2024-01-15 12:00:00', 'GMT+1')` already returns 
`2024-01-15T13:00:00`.
   
   ### Expected behavior
   
   The same results as Spark, which resolves the local time with 
`LocalDateTime.atZone`. `TimeZoneRules::local_to_utc` in `datetime/timezone.rs` 
already does that, including Spark's handling of gaps and overlaps, and is the 
inverse of the `adjust_to_local_time` that `from_utc_timestamp` uses. Parsing 
the zone with `TimeZoneRules::parse` would also accept the zone IDs that 
`from_utc_timestamp` accepts.
   
   ### Additional context
   
   Found while reviewing for #353, which tracks moving functions onto the 
shared cast and time zone code.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to