Hyukjin Kwon created SPARK-59101:
------------------------------------
Summary: Release the Rust Spark Connect client from a chosen
apache upstream branch on manual dispatch
Key: SPARK-59101
URL: https://issues.apache.org/jira/browse/SPARK-59101
Project: Spark
Issue Type: Improvement
Components: Project Infra
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The Rust-backed Spark Connect client (apache/spark-connect-rust) drop-in
under-exposes
several public pyspark v4.2.0 APIs. A rigorous introspection-based audit of the
v4.2.0
pyspark.sql.connect.* surface vs the current drop-in found the following gaps,
to be
closed while mirroring the official pyspark implementation exactly and keeping
test coverage:
Layer A (Rust core already implements it; only Python exposure missing):
- functions higher-order: transform, filter, exists, forall, aggregate, reduce,
zip_with,
transform_keys, transform_values, map_filter, map_zip_with
- functions misc: cume_dist, broadcast, call_function, call_udf, column
- DataFrame.repartitionById; RuntimeConfig.getAll
- DataType/StructType/StructField: json, jsonValue, simpleString, typeName,
fieldNames, fromDDL
Layer B (missing in both core and Python):
- Catalog: analyzeTable, createDatabase, dropDatabase, dropTable, dropView,
getCreateTableString, getTableProperties, listPartitions, listViews,
truncateTable
- types model: toDDL, treeString, toNullable, fromJson, fromInternal,
toInternal, needConversion
- DataFrame.zipWithIndex; Column.transform; DataFrameStatFunctions.sampleBy
- DataFrameReader.changes; DataStreamReader.changes; DataStreamReader.xml;
StreamingQueryManager.close
- functions: udtf, arrow_udtf, arrow_udf
- UDFRegistration.registerJavaFunction, registerJavaUDAF
- ml.connect base classes / param mixins / *Model / ArrayAssembler /
TorchDistributor
- Row.count, Row.index
Module structure: expose pyspark.sql.avro.functions,
pyspark.sql.protobuf.functions,
pyspark.sql.functions.partitioning submodules (functions currently exist only
flat).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
