[
https://issues.apache.org/jira/browse/SPARK-61118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61118:
---------------------------------
Description:
The {{pyspark-client}} package of Apache Spark v4.3.0 ships the CLI of Spark
Declarative Pipelines: {{pyspark.pipelines.cli}} ({{spark-pipelines run}},
{{dry-run}} and {{init}}), {{init_cli}}, {{add_pipeline_analysis_context}} and
{{block_session_mutations}}. The Python client lacks these modules, and
importing them raises {{ModuleNotFoundError}}.
{{add_pipeline_analysis_context}} adds a user context extension to the requests
of a thread ({{SparkConnectClient.add_threadlocal_user_context_extension}} and
{{remove_user_context_extension}}), which the Rust transport needs to send.
The fix is to vendor these modules, with their official tests, from Apache
Spark v4.3.0, and to add the user context extensions to the client:
{{add_threadlocal_user_context_extension}},
{{add_global_user_context_extension}}, {{remove_user_context_extension}},
{{clear_user_context_extensions}}, {{thread_local}} and
{{global_user_context_extensions}}, which the Rust transport sends with the
requests of executions, analyses, configurations, error details and interrupts,
as PySpark's {{SparkConnectClient}} does.
was:
The {{pyspark-client}} package of Apache Spark v4.3.0 ships the CLI of Spark
Declarative Pipelines: {{pyspark.pipelines.cli}} ({{spark-pipelines run}},
{{dry-run}} and {{init}}), {{init_cli}}, {{add_pipeline_analysis_context}} and
{{block_session_mutations}}. The Python client lacks these modules, and
importing them raises {{ModuleNotFoundError}}.
{{add_pipeline_analysis_context}} adds a user context extension to the requests
of a thread ({{SparkConnectClient.add_threadlocal_user_context_extension}} and
{{remove_user_context_extension}}), which the Rust transport needs to send.
Summary: Add the CLI modules of pyspark.pipelines, and the user context
extensions of the client, in the Python client as PySpark has them (was: Add
the CLI modules of pyspark.pipelines, in the Python client as PySpark has them)
> Add the CLI modules of pyspark.pipelines, and the user context extensions of
> the client, in the Python client as PySpark has them
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61118
> URL: https://issues.apache.org/jira/browse/SPARK-61118
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The {{pyspark-client}} package of Apache Spark v4.3.0 ships the CLI of Spark
> Declarative Pipelines: {{pyspark.pipelines.cli}} ({{spark-pipelines run}},
> {{dry-run}} and {{init}}), {{init_cli}}, {{add_pipeline_analysis_context}}
> and {{block_session_mutations}}. The Python client lacks these modules, and
> importing them raises {{ModuleNotFoundError}}.
> {{add_pipeline_analysis_context}} adds a user context extension to the
> requests of a thread
> ({{SparkConnectClient.add_threadlocal_user_context_extension}} and
> {{remove_user_context_extension}}), which the Rust transport needs to send.
> The fix is to vendor these modules, with their official tests, from Apache
> Spark v4.3.0, and to add the user context extensions to the client:
> {{add_threadlocal_user_context_extension}},
> {{add_global_user_context_extension}}, {{remove_user_context_extension}},
> {{clear_user_context_extensions}}, {{thread_local}} and
> {{global_user_context_extensions}}, which the Rust transport sends with the
> requests of executions, analyses, configurations, error details and
> interrupts, as PySpark's {{SparkConnectClient}} does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
