Jason Chen created SPARK-60161:
----------------------------------
Summary: Add tests for nested timestamp conversion in pandas type
converters
Key: SPARK-60161
URL: https://issues.apache.org/jira/browse/SPARK-60161
Project: Spark
Issue Type: Test
Components: PySpark, Tests
Affects Versions: 4.4.0
Reporter: Jason Chen
The pandas type converters in {{pyspark/sql/pandas/types.py}}
({{_create_converter_to_pandas}} and {{_create_converter_from_pandas}}) handle
top-level timestamp columns with a vectorized path, but timestamps nested
inside complex types (e.g. {{ArrayType(TimestampType())}}) go through per-value
converters that currently lack unit test coverage.
According to Codecov, several of these branches are not exercised by any test,
including:
* from_pandas: timezone-aware datetime values nested in {{TimestampType}}
* from_pandas: {{TimestampLTZNanosType}} values (the whole converter)
* to_pandas: nested {{TimestampType}} with {{timestamp_utc_localized=False}}
* to_pandas: nested {{TimestampLTZNanosType}} and {{TimestampNTZType}} values
This ticket adds unit tests to
{{python/pyspark/sql/tests/pandas/test_converter.py}} covering both directions
for nested {{TimestampType}}, {{TimestampLTZNanosType}} and
{{TimestampNTZType}}, including:
* timezone-aware vs. naive input values
* interpretation of naive values in the session timezone
* {{timestamp_utc_localized=False}}, where naive values are read in the local
(system) timezone
* preservation of nanosecond precision for {{TimestampLTZNanosType}}
* None elements and None arrays
This is a test-only change with no user-facing impact.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
