[
https://issues.apache.org/jira/browse/SPARK-59109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-59109.
----------------------------------
Fix Version/s: connect-rust-4.2.0
connect-rust-4.3.0
Resolution: Fixed
> Fix PySpark API-surface parity in the Rust drop-in: properties, show()
> default, camelCase kwargs, __version__
> -------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59109
> URL: https://issues.apache.org/jira/browse/SPARK-59109
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.2.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.2.0, connect-rust-4.3.0
>
>
> An audit of the Rust-backed drop-in against real PySpark 4.2.0 found basic
> API-surface parity gaps that the transport-level official-suite gate never
> exercises (it runs upstream PySpark over the Rust transport, not over this
> Python API surface).
> - pyspark.__version__ was "0.1.0"; now sourced from pyspark.version ("4.2.0").
> - SparkSession.version and DataFrame.dtypes / executionInfo were methods;
> exposed as properties to match PySpark attribute access.
> - DataFrame.show() had a required n; now defaults to (n=20, truncate=True,
> vertical=False).
> - ~30 PyO3 method parameters were snake_case; renamed to PySpark camelCase so
> keyword calls match (e.g. sql(sqlQuery), cast(dataType), limit(num),
> withColumn(colName), between(lowerBound/upperBound)).
> Adds offline introspection guards; verified to zero diffs against a real
> PySpark 4.2.0 signature/kind audit and against a live Spark Connect server.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
