cj-zhukov commented on PR #86:
URL: https://github.com/apache/datafusion-spark/pull/86#issuecomment-6922396021

   > Thanks @cj-zhukov, this addresses everything from my last review. 
Timestamps now use the session time zone and ms precision. Floats and decimals 
follow the Java toString rules. Map keys use Spark's internal values. 
Unsupported types are rejected at planning time with useful paths. The slt file 
is thorough and the API surface is down to `export_functions!`. I spot-checked 
several extra cases against Spark 4.2.0 (`1e7` -> `1.0E7`, `9.999e-4` -> 
`9.999E-4`, `-0.0`, NTZ/TZ timestamps with whole seconds -> `.000`, NTZ map 
keys as micros, `-1.0E-7` decimals) and they all match.
   > 
   > The two deviations you called out are fine with me: lowercase escapes in 
struct field names, and rejecting struct/list map keys.
   > 
   > One small nit, which can be a follow-up: `%:z` rounds offsets to the 
minute, so historical LMT offsets come out wrong. With session TZ 
`America/Los_Angeles`, Spark writes `TIMESTAMP'1800-01-01 00:00:00'` as 
`1800-01-01T00:00:00.000-07:52:58`, and this PR would write `-07:53`. Using 
`%::z` when the offset has a seconds part would fix it.
   > 
   > Could you also open an issue for map/array input and the options argument?
   > 
   > I've approved the CI workflows. LGTM once they pass.
   
   @andygrove Thanks @andygrove. I'll open two follow-up issues: one for the 
`%::z` offset fix and one for map/array input and the options argument. I'll 
link them here once they're up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to