Hyukjin Kwon created SPARK-61128:
------------------------------------
Summary: Make ArrayType, MapType, StructField and StructType give
back and pickle the user-defined types they are made of, and StructField
picklable, in the Python client as PySpark does
Key: SPARK-61128
URL: https://issues.apache.org/jira/browse/SPARK-61128
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's ArrayType, MapType, StructField and StructType hold the type objects
they are made of. So ArrayType(udt).elementType is udt, a user-defined type,
and so are MapType(StringType(), udt).valueType, StructField("p", udt).dataType
and the dataType of the fields of a StructType. A type pickles of those
objects, and a user-defined type pickles by its class.
The Python client of spark-connect-rust lowers these types into its Rust core,
and rebuilds a user-defined type from its JSON, by importing its pyClass,
whenever it gives the type back. The pyClass need not name the module of the
class: for Apache Spark's ExamplePointUDT, it is
pyspark.sql.tests.ExamplePointUDT, while the class is in
pyspark.testing.objects. So on the client:
- ArrayType(ExamplePointUDT()).elementType, and the types of the other
containers above, raise PySparkValueError UNSUPPORTED_OPERATION ("unpickling
user defined types is not supported")
- so do their repr and simpleString, and
udf(returnType=ArrayType(ExamplePointUDT())), which Apache Spark's
test_udf_with_udt calls
- the pickle of such a type, by its JSON, fails the same way on the server's
Python worker
Also, a StructField of the client cannot be pickled at all ("TypeError: cannot
pickle 'pyspark.sql.types.StructField' object"), where PySpark's can.
The client's types should give back the user-defined types they are made of,
and pickle with them, and its StructField should pickle, as PySpark's do.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
