Varun Bhandary created SPARK-59373:
--------------------------------------
Summary: Fix the documented return type of ceil and floor with a
scale argument
Key: SPARK-59373
URL: https://issues.apache.org/jira/browse/SPARK-59373
Project: Spark
Issue Type: Documentation
Components: Documentation
Affects Versions: 5.0.0, 4.4.0
Reporter: Varun Bhandary
The Returns block in pyspark.sql.functions.floor contradicts itself. It says
"nearest integer that is less than or equal to given value" and then, on the
very
next line, "Returns a column that evaluates to a long or decimal".
The first line is a leftover from before SPARK-45131 and is wrong whenever scale
is passed. The docstring's own example disproves it: floor(2.1267, 2) returns
Decimal('2.12'), which is not an integer.
The second line is accurate but does not say which case gives which, and the
rule
is not obvious, because passing scale returns a decimal for every input type,
integers included. Measured on 4.0.0:
floor(double) is bigint, and floor(double, 1) is decimal(17,1)
ceil(double) is bigint, and ceil(double, 1) is decimal(17,1)
round(double, 1) is double
floor(bigint, 1) is decimal(21,0)
So ceil and floor are the only scale taking math functions that do not return
the
input's type, and nothing in the documentation says so. round, bround and
truncate all preserve it.
This is a documentation fix rather than a behaviour change. SPARK-38604 records
that these return types were chosen deliberately when SPARK-37475 added the
scale parameter.
The fix is to remove the wrong line, state the rule in the Python and Scala docs
for floor, ceil and ceiling, and add it to the ceil and floor SQL function
descriptions, which currently say nothing about the return type at all.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
