[
https://issues.apache.org/jira/browse/SPARK-61155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61155:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Unpickle the DataFrames that foreachBatch functions capture on the Python
> workers of the server, in the Python client as PySpark does
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61155
> URL: https://issues.apache.org/jira/browse/SPARK-61155
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> A foreachBatch function runs on a Python worker of the server, which has
> PySpark and an active session of the query. A DataFrame that the function
> captures is pickled with it: PySpark Connect's unpickles there, of that
> session and of its plan.
> The Python client pickles a DataFrame as a function, by value, that rebuilds
> it with the Rust client (pyspark._pyspark), which the worker does not have:
> StreamingPythonRunnerInitializationException:
> [STREAMING_PYTHON_RUNNER_INITIALIZATION_FAILURE] ... ImportError: cannot
> import name '_pyspark' from 'pyspark'
> So any foreachBatch function that uses a DataFrame defined outside of it
> fails to start. Four tests of Apache Spark's test_parity_foreach_batch.py
> fail on this client: test_accessing_spark_session_through_df,
> test_nested_dataframes,
> test_batch_dataframe_join_with_root_session_dataframe_is_unsupported and
> test_disabling_fix_restores_root_session_behavior.
> On a worker without the Rust client, the function should give back PySpark
> Connect's DataFrame of the relation, of its plan ID.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
