[
https://issues.apache.org/jira/browse/SPARK-61151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61151:
------------------------------------
Assignee: Hyukjin Kwon
> Vendor pyspark.sql.streaming.listener, of the listener and its event and
> progress classes, in the Python client as PySpark has it
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61151
> URL: https://issues.apache.org/jira/browse/SPARK-61151
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The Python client's pyspark.sql.streaming.listener is its own rewrite of
> PySpark's, which differs from PySpark 4.3.0's:
> - StreamingQueryProgress.numInputRows, inputRowsPerSecond and
> processedRowsPerSecond are None for the progress of the server, whose JSON
> leaves them out. PySpark's sums those of the sources then. So
> q.lastProgress.numInputRows and q.recentProgress[i].numInputRows are None.
> - StreamingQueryListener has no spark property and no _set_spark_session.
> - __all__ lists the events and progress classes, where PySpark's has only
> StreamingQueryListener.
> The module is pure Python and needs nothing of the client: it should be
> PySpark's, vendored verbatim and checked by the drift check. Apache Spark's
> test_batch_dataframe_in_batch_session_sql_is_supported and
> test_batch_dataframe_join_in_batch_session_is_supported of
> test_parity_foreach_batch.py fail on this client ('>' not supported between
> instances of 'NoneType' and 'int').
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
