[
https://issues.apache.org/jira/browse/SPARK-57696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-57696:
-----------------------------------
Labels: pull-request-available (was: )
> Preserve TIME precision in the PySpark Arrow/pandas type mapping
> ----------------------------------------------------------------
>
> Key: SPARK-57696
> URL: https://issues.apache.org/jira/browse/SPARK-57696
> Project: Spark
> Issue Type: Sub-task
> Components: Pandas API on Spark
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
> Labels: pull-request-available
>
> h2. What
> Carry the {{TimeType(p)}} fractional-second precision {{p}} (in [0, 9])
> across the
> PySpark Arrow/pandas type mapping in {{python/pyspark/sql/pandas/types.py}}
> so that a
> {{TIME(p)}} column round-trips back to the same {{TIME(p)}}, instead of
> collapsing to the
> default {{TimeType()}} (precision 6).
> This is the PySpark counterpart of SPARK-57661, which fixes the same gap in
> the JVM
> {{ArrowUtils}} (Spark <-> Arrow) mapping.
> h2. Why
> {{to_arrow_type}} maps {{TimeType}} to {{pa.time64("ns")}}, which carries
> only a time unit
> and no fractional-second precision field. {{from_arrow_type}} maps any
> {{is_time64(at)}}
> back to {{TimeType()}}, whose constructor defaults to {{precision = 6}}. As a
> result the
> declared precision is lost on any Arrow/pandas round-trip ({{TIME(0)}},
> {{TIME(3)}},
> {{TIME(9)}}, ... all read back as {{TIME(6)}}), so PySpark Arrow-based
> transfer
> (createDataFrame from Arrow/pandas, {{mapInArrow}}, {{toPandas}} schema,
> etc.) silently
> widens or narrows the type label. The stored values are unaffected; this is
> purely a
> type-fidelity gap.
> PyArrow's {{time64}} logical type only encodes the unit (us/ns) and cannot
> represent
> precision [0, 9], so the precision must be carried in the Arrow field
> metadata, mirroring
> the {{SPARK::time::precision}} key the JVM {{ArrowUtils}} uses in SPARK-57661.
> h2. Scope
> * {{python/pyspark/sql/pandas/types.py}}: when building the Arrow field for a
> {{TimeType(p)}},
> tag it with the precision metadata key {{SPARK::time::precision}}; when
> reading an Arrow
> {{time64}} field back, read that key to reconstruct {{TimeType(p)}}.
> * Reuse the same metadata key and fallback semantics as the JVM side
> (SPARK-57661) for
> consistency.
> h2. Behavior on read-back
> * Metadata present and in [0, 9]: reconstruct the exact {{TimeType(p)}}.
> * Metadata absent (foreign Arrow data) or out of [0, 9]: fall back to the
> default
> {{TimeType()}} (precision 6), preserving today's behavior for non-Spark
> producers.
> h2. Out of scope
> * Value semantics / rounding: values are carried verbatim; no change to how
> {{TIME(p)}}
> values are truncated.
> * The JVM {{ArrowUtils}} mapping (handled in SPARK-57661) and the Spark
> Connect
> proto/converters (separate sub-tasks).
> h2. How tested
> PySpark Arrow conversion tests: round-trip {{TIME(p)}} for {{p}} in {0, 3, 6,
> 9} preserves
> {{p}}; a {{time64}} field with no precision metadata falls back to
> {{TIME(6)}}; the
> precision key does not leak into reconstructed field metadata.
> h2. Does this introduce any user-facing change
> No. The TIME data type is gated behind the internal flag
> {{spark.sql.timeType.enabled}},
> which is off by default in production. With the flag enabled, a {{TIME(p)}}
> column
> transferred over Arrow/pandas in PySpark retains its declared precision
> instead of always
> reading back as {{TIME(6)}}. No change to stored values.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
