[
https://issues.apache.org/jira/browse/SPARK-61133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61133:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Add the private helpers of pyspark.sql.types, such as _infer_schema,
> _merge_type and _make_type_verifier, in the Python client as PySpark has them
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61133
> URL: https://issues.apache.org/jira/browse/SPARK-61133
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark's pyspark.sql.types has private helpers that PySpark Connect's
> modules, Apache Spark's tests and some libraries import:
> - functions: _infer_type, _infer_schema, _merge_type, _has_nulltype,
> _need_converter, _create_converter, _make_type_verifier,
> _create_row_inbound_converter and _int_size_to_type
> - tables: _type_mappings, _array_type_mappings,
> _array_signed_int_typecode_ctype_mappings,
> _array_unsigned_int_typecode_ctype_mappings, _acceptable_types,
> _atomic_types, _complex_types, _all_complex_types, _all_mappable_types and
> JVM_INT_MAX
> PySpark Connect's session.py imports _infer_schema, _has_nulltype and
> _merge_type. Apache Spark's test_types.py, run by test_parity_types.py,
> imports _infer_type, _make_type_verifier, _merge_type and the typecode
> mappings.
> The Python client of spark-connect-rust lacks all of them. So "from
> pyspark.sql.types import _merge_type" raises ImportError, and
> test_parity_types.py cannot be collected on this client.
> The client should have them, as PySpark's pure-Python implementations of
> these types, which the client's types share the classes and attributes of.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
