Hyukjin Kwon created SPARK-61133:
------------------------------------
Summary: Add the private helpers of pyspark.sql.types, such as
_infer_schema, _merge_type and _make_type_verifier, in the Python client as
PySpark has them
Key: SPARK-61133
URL: https://issues.apache.org/jira/browse/SPARK-61133
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's pyspark.sql.types has private helpers that PySpark Connect's modules,
Apache Spark's tests and some libraries import:
- functions: _infer_type, _infer_schema, _merge_type, _has_nulltype,
_need_converter, _create_converter, _make_type_verifier,
_create_row_inbound_converter and _int_size_to_type
- tables: _type_mappings, _array_type_mappings,
_array_signed_int_typecode_ctype_mappings,
_array_unsigned_int_typecode_ctype_mappings, _acceptable_types, _atomic_types,
_complex_types, _all_complex_types, _all_mappable_types and JVM_INT_MAX
PySpark Connect's session.py imports _infer_schema, _has_nulltype and
_merge_type. Apache Spark's test_types.py, run by test_parity_types.py, imports
_infer_type, _make_type_verifier, _merge_type and the typecode mappings.
The Python client of spark-connect-rust lacks all of them. So "from
pyspark.sql.types import _merge_type" raises ImportError, and
test_parity_types.py cannot be collected on this client.
The client should have them, as PySpark's pure-Python implementations of these
types, which the client's types share the classes and attributes of.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
