[
https://issues.apache.org/jira/browse/SPARK-59571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-59571.
------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 58850
[https://github.com/apache/spark/pull/58850]
> Cover every TIME precision, null pattern and interval sign in
> ArrowCachedBatchSerializerSuite
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-59571
> URL: https://issues.apache.org/jira/browse/SPARK-59571
> Project: Spark
> Issue Type: Sub-task
> Components: SQL, Tests
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> h3. What
> Add two tests to \{{ArrowCachedBatchSerializerSuite}} for the Arrow cache
> serializer (SPARK-57268) over the \{{TIME}} and \{{INTERVAL DAY TO SECOND}}
> types:
> * \{{TIME(p)}} for p in 0, 3, 6 and 9, each at three null patterns (every
> 31st row null, no nulls, all nulls), a thousand rows in one partition. Each
> case checks that the cached frame answers what the uncached one did, that the
> cached \{{InMemoryRelation}} keeps \{{TimeType(p)}} with the declared
> precision, and that the serializer's own columnar read path yields a
> \{{TimeNanoVector}} per batch whose row and null counts add up to the
> input's. The values are truncated to the declared precision so a precision
> loss could not hide behind a value the type would round anyway.
> * \{{INTERVAL DAY TO SECOND}} with whole microseconds of both signs, at the
> same three null patterns, checking the same three things against a
> \{{DurationVector}}.
> h3. Why
> The suite today round-trips \{{TIME}} once, at precision 6, over two non-null
> rows, and checks its \{{LongColumnStats}}. Every precision of \{{TIME}} is
> written to the same \{{TimeNanoVector}}, and the precision travels only in
> the Arrow field metadata, so a regression that dropped it on the way back
> through the cache would leave the values intact and fail no existing test.
> The all-null column takes a branch of its own in the vector (no validity
> buffer), and negative intervals are not covered at all. \{{TIME}} is about to
> be enabled by default, and the Arrow cache is one of the paths it takes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
