Hyukjin Kwon created SPARK-57999:
------------------------------------
Summary: Add a generated, test-backed table of accepted input
types for built-in scalar functions
Key: SPARK-57999
URL: https://issues.apache.org/jira/browse/SPARK-57999
Project: Spark
Issue Type: Test
Components: SQL, Tests
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Spark's built-in functions do not consistently document which input DataTypes
each argument accepts. The authoritative constraint lives only on the catalyst
expression (inputTypes / checkInputDataTypes), and the public functions APIs
(functions.scala, pyspark.sql.functions) are name-based facades over it.
As a foundation for documenting accepted types (and per the principle that we
should not document what is not tested), this adds FunctionAcceptedTypesSuite:
an exhaustive, generated suite that, for every built-in scalar function and
every argument position, probes each candidate DataType through the analyzer
and records whether it is accepted (declared natively via ExpectsInputTypes vs.
only via implicit cast), rejected (DATATYPE_MISMATCH), or inconclusive (other
analysis errors). Results are written to a golden file,
sql/core/src/test/resources/sql-functions/sql-function-accepted-types.md,
regenerated with SPARK_GENERATE_GOLDEN_FILES=1.
The suite is tagged @ExtendedSQLTest so it is skipped in the default test run
(it probes every function against every type) and runs only in the dedicated
extended-tests CI leg.
This is the first step; follow-ups will use the golden table to drive and
verify accepted-type documentation in SQL @ExpressionDescription, Scala
Scaladoc, and PySpark docstrings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
