Anas Khan created SPARK-57965:
---------------------------------
Summary: UNSUPPORTED_PIPELINES_DATASET_TYPE raises AssertionError
due to wrong message parameter name
Key: SPARK-57965
URL: https://issues.apache.org/jira/browse/SPARK-57965
Project: Spark
Issue Type: Bug
Components: Connect, PySpark
Affects Versions: 4.1.0
Reporter: Anas Khan
SparkConnectGraphElementRegistry.register_output
(python/pyspark/pipelines/spark_connect_graph_element_registry.py) raises
PySparkTypeError with error class UNSUPPORTED_PIPELINES_DATASET_TYPE and
messageParameters key `output_type` at two else-branch raise sites.
The UNSUPPORTED_PIPELINES_DATASET_TYPE template in
python/pyspark/errors/error-conditions.json interpolates <dataset_type>:
"Unsupported pipelines dataset type: <dataset_type>."
ErrorClassesReader.get_error_message (python/pyspark/errors/utils.py) asserts
the template placeholders equal the provided message parameters. Since
{dataset_type} != \{output_type}, the assertion fires inside
PySparkException.__init__ and an opaque AssertionError replaces the intended
PySparkTypeError.
Fix: rename the messageParameters key `output_type` -> `dataset_type` at both
raise sites (the value, type(output).__name__, is unchanged) and add a
regression test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
