Hyukjin Kwon created SPARK-60981:
------------------------------------
Summary: Alias a Column with several names, and with metadata of
any JSON values, in Spark Connect Rust as PySpark does
Key: SPARK-60981
URL: https://issues.apache.org/jira/browse/SPARK-60981
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark, {{Column.alias(*alias, metadata=None)}} names a column with each
given name, so a generator's columns can all be named:
{{F.explode(F.col("m")).alias("k", "v")}} names the key and the value of a map.
The metadata is any JSON-serializable dict, which is sent as
{{json.dumps(metadata)}}, and metadata with more than one name raises
{{PySparkValueError}} {{ONLY_ALLOWED_FOR_SINGLE_COLUMN}}. The Rust-backed
Python client differs:
* It keeps only the first name, so {{spark.sql("SELECT map('k1', 1) AS
m").select(F.explode("m").alias("k", "v"))}} fails on the server with
{{UDTF_ALIAS_NUMBER_MISMATCH}}, where PySpark 4.3.0 gives the columns {{k}} and
{{v}}.
* It takes metadata of string values only, so {{F.col("id").alias("x",
metadata={"k": 1})}} raises {{TypeError: 'int' object is not an instance of
'str'}}.
The Rust {{Column}} has no alias of several names either. Alias with several
names, and with metadata of any JSON values, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
