[
https://issues.apache.org/jira/browse/SPARK-57742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092160#comment-18092160
]
Jubin Soni commented on SPARK-57742:
------------------------------------
I'm working on this
> unix_micros docstring is missing the truncation disclaimer present in
> unix_millis and unix_seconds
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-57742
> URL: https://issues.apache.org/jira/browse/SPARK-57742
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.3.0, 5.0.0
> Reporter: Jubin Soni
> Priority: Trivial
>
> *What is the issue?*
> Within the {{unix_*}} epoch-unit function family, {{unix_millis}} and
> {{unix_seconds}} both document that they truncate sub-unit precision.
> {{unix_micros}} does not, despite having the same behaviour when given a
> nanosecond-precision input.
> Current one-line descriptions:
> {{unix_seconds → "Returns the number of seconds since 1970-01-01 00:00:00
> UTC. Truncates higher levels of precision."
> unix_millis → "Returns the number of milliseconds since 1970-01-01 00:00:00
> UTC. Truncates higher levels of precision."
> unix_micros → "Returns the number of microseconds since 1970-01-01 00:00:00
> UTC." ← missing disclaimer}}
> This became a practical problem once {{unix_nanos}} (SPARK-57579) made
> nanosecond-precision timestamps usable: a user holding a {{TIMESTAMP_LTZ(9)}}
> or {{TIMESTAMP_NTZ(9)}} column who reads the {{unix_micros}} docstring has no
> indication that sub-microsecond digits will be silently dropped.
> ----
> *How to reproduce*
> {{spark.conf.set("spark.sql.timestampNanosTypes.enabled", "true")
> df = spark.sql("SELECT TIMESTAMP_NTZ '2020-01-01 00:00:00.000000999' AS ts")
> from pyspark.sql import functions as sf
> # unix_micros silently drops the 999 nanoseconds — nothing in the docstring
> warns of this
> df.select(sf.unix_micros("ts")).show()}}
> ----
> *Actual behavior*
> The docstring for {{unix_micros}} contains no truncation warning, yet:
> {{+--------------------+
> |unix_micros(ts) |
> +--------------------+
> |1577836800000000 | ← 999 nanoseconds silently dropped
> +--------------------+}}
> ----
> *Expected behavior*
> The docstring reads:
> {{"Returns the number of microseconds since 1970-01-01 00:00:00 UTC.
> Truncates higher levels of precision."}}
> matching {{unix_millis}} and {{{}unix_seconds{}}}, so users are warned that
> sub-microsecond digits are dropped.
> ----
> *Proposed fix*
> Add {{"Truncates higher levels of precision."}} to the one-line description
> of {{unix_micros}} in {{{}python/pyspark/sql/functions/builtin.py{}}}. One
> line change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
