[
https://issues.apache.org/jira/browse/SPARK-61115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61115:
------------------------------------
Assignee: Hyukjin Kwon
> Add the module-level names and the connect modules that the Python client
> lacks, as PySpark 4.3.0 has them
> ----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61115
> URL: https://issues.apache.org/jira/browse/SPARK-61115
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The {{pyspark-client}} package of Apache Spark v4.3.0 ({{connect_packages}}
> of {{python/packaging/client/setup.py}}) has module-level names and modules
> that the Python client lacks:
> * {{pyspark.util}}: {{fail_on_stopiteration}}, {{disable_gc}},
> {{spark_connect_mode}}, {{default_api_mode}}, {{FuncT}} and its {{__all__}}.
> * The {{__all__}} of {{pyspark.sql.udf}}, {{pyspark.sql.udtf}},
> {{pyspark.sql.avro}} and {{pyspark.sql.protobuf}}. Without them, {{from
> pyspark.sql.udtf import *}} exports the other public names of the module too.
> * {{pyspark.sql.functions.builtin}} and
> {{pyspark.sql.connect.functions.builtin}} lack {{approxCountDistinct}},
> {{bitwiseNOT}}, {{countDistinct}}, {{shiftLeft}}, {{shiftRight}},
> {{shiftRightUnsigned}}, {{sumDistinct}}, {{toDegrees}}, {{toRadians}},
> {{chr}}, {{random}} and {{uuid}}, which {{pyspark.sql.functions}} has. They
> also have an {{__all__}}, which PySpark's {{builtin}} does not. So {{from
> pyspark.sql.functions.builtin import countDistinct}} raises {{ImportError}}.
> * The connect import path lacks
> {{pyspark.sql.connect.functions.partitioning}},
> {{pyspark.sql.connect.avro.functions}},
> {{pyspark.sql.connect.protobuf.functions}},
> {{pyspark.sql.connect.table_arg}}, {{pyspark.sql.connect.datasource}},
> {{pyspark.sql.connect.utils}}, {{pyspark.sql.connect.logging}} and
> {{pyspark.sql.connect._typing}}. Importing them raises
> {{ModuleNotFoundError}}.
> The fix is to add them, as PySpark has them. Pure-Python modules are
> vendored. The connect modules re-export the functions and classes of the
> client, and {{pyspark.sql.connect.utils.check_dependencies}} checks the
> dependencies of this client, which do not include gRPC.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
