alamb commented on code in PR #26867: URL: https://github.com/apache/datafusion/pull/26867#discussion_r4415695871
########## datafusion/sqllogictest/test_files/nested_loop_join_spill.slt: ########## @@ -302,7 +302,7 @@ RIGHT SEMI JOIN existence_spill_right r ON l.threshold > r.k Plan with Metrics 01)ProjectionExec: expr=[count(Int64(1))@0 as count(*)], metrics=[<slt:ignore>] 02)--AggregateExec: mode=Single, gby=[], aggr=[count(Int64(1))], metrics=[<slt:ignore>] -03)----NestedLoopJoinExec: join_type=RightSemi, filter=threshold@0 > k@1, projection=[], metrics=[output_rows=128, <slt:ignore> spill_count=2, <slt:ignore> input_batches=8, input_rows=516, <slt:ignore>] +03)----NestedLoopJoinExec: join_type=RightSemi, filter=threshold@0 > k@1, projection=[], metrics=[output_rows=128, <slt:ignore> spill_count=2, <slt:ignore> input_batches=6, input_rows=387, <slt:ignore>] Review Comment: Due to - https://github.com/apache/arrow-rs/pull/11694 The right side is replayed once per left chunk. Spilled batches are now read back without validity buffers for null-free columns, so they take less memory and the build side fits in three chunks instead of four. ########## datafusion/sqllogictest/test_files/aggregate.slt: ########## @@ -1732,7 +1732,7 @@ SELECT approx_distinct(c9) AS a, approx_distinct(c9) AS b FROM aggregate_test_10 query IIIII SELECT approx_distinct(c14) AS a, approx_distinct(c15) AS b, approx_distinct(arrow_cast(c15, 'Date64')), approx_distinct(arrow_cast(c15, 'Time32(Second)')) as c, approx_distinct(arrow_cast(c15, 'Time64(Nanosecond)')) AS d FROM aggregate_test_100 ---- -18 60 60 60 60 +18 60 7 60 60 Review Comment: Due to - https://github.com/apache/arrow-rs/pull/11632 Casting a Timestamp to Date64 now truncates to whole days (previously it kept the time of day), so only 7 distinct dates remain. ########## datafusion/sqllogictest/test_files/nested_loop_join_spill.slt: ########## @@ -294,6 +294,9 @@ RIGHT ANTI JOIN existence_spill_right r ON l.threshold > r.k ---- 1064 NULL payload-64 +# The right side is replayed once per left chunk. Spilled batches are read back Review Comment: Due to - https://github.com/apache/arrow-rs/pull/11694 ########## datafusion/sqllogictest/test_files/datetime/date_part.slt: ########## @@ -1069,12 +1069,14 @@ FROM (SELECT arrow_cast(column1, 'Timestamp(Nanosecond, Some("Africa/Monrovia")) 28 28500 28500000 28500000000 NULL NULL NULL NULL -# timestamps outside the range arrow represents are NULL, like the other parts +# The second is computed directly from the stored value (i64::MAX % 60 = 7). The +# sub-second parts of a timestamp outside the range arrow represents are NULL, +# like the other parts. query IIII SELECT extract(second from ts), extract(millisecond from ts), extract(microsecond from ts), extract(nanosecond from ts) FROM (VALUES (arrow_cast(9223372036854775807, 'Timestamp(Second, None)')), (NULL)) AS t(ts) ---- -NULL NULL NULL NULL +7 NULL NULL NULL Review Comment: Due to - https://github.com/apache/arrow-rs/pull/11187 `second` is now computed directly from the stored value for timestamps without a timezone, so `i64::MAX` seconds yields `7` (`i64::MAX % 60`) instead of overflowing to `NULL`. The sub-second parts still go through chrono and remain `NULL`. The nullability reproducer further down now uses `millisecond` so it still produces a `NULL` from a non-null input. ########## datafusion/sqllogictest/test_files/aggregate.slt: ########## @@ -1729,10 +1729,12 @@ SELECT approx_distinct(c9) AS a, approx_distinct(c9) AS b FROM aggregate_test_10 99 99 # csv_query_approx_count_date_timestamp +# Casting a Timestamp to Date64 truncates to whole days (arrow-rs 60.1.0), so Review Comment: Due to - https://github.com/apache/arrow-rs/pull/11632 ########## datafusion/sqllogictest/test_files/datetime/date_part.slt: ########## @@ -1069,12 +1069,14 @@ FROM (SELECT arrow_cast(column1, 'Timestamp(Nanosecond, Some("Africa/Monrovia")) 28 28500 28500000 28500000000 NULL NULL NULL NULL -# timestamps outside the range arrow represents are NULL, like the other parts +# The sub-second parts of a timestamp outside the range arrow represents are Review Comment: Due to - https://github.com/apache/arrow-rs/pull/11187 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
