[
https://issues.apache.org/jira/browse/SPARK-59101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-59101:
------------------------------------
Assignee: Hyukjin Kwon
> Release the Rust Spark Connect client from a chosen apache upstream branch on
> manual dispatch
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-59101
> URL: https://issues.apache.org/jira/browse/SPARK-59101
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Affects Versions: connect-rust-4.2.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The Rust-backed Spark Connect client (apache/spark-connect-rust) drop-in
> under-exposes
> several public pyspark v4.2.0 APIs. A rigorous introspection-based audit of
> the v4.2.0
> pyspark.sql.connect.* surface vs the current drop-in found the following
> gaps, to be
> closed while mirroring the official pyspark implementation exactly and
> keeping test coverage:
> Layer A (Rust core already implements it; only Python exposure missing):
> - functions higher-order: transform, filter, exists, forall, aggregate,
> reduce, zip_with,
> transform_keys, transform_values, map_filter, map_zip_with
> - functions misc: cume_dist, broadcast, call_function, call_udf, column
> - DataFrame.repartitionById; RuntimeConfig.getAll
> - DataType/StructType/StructField: json, jsonValue, simpleString, typeName,
> fieldNames, fromDDL
> Layer B (missing in both core and Python):
> - Catalog: analyzeTable, createDatabase, dropDatabase, dropTable, dropView,
> getCreateTableString, getTableProperties, listPartitions, listViews,
> truncateTable
> - types model: toDDL, treeString, toNullable, fromJson, fromInternal,
> toInternal, needConversion
> - DataFrame.zipWithIndex; Column.transform; DataFrameStatFunctions.sampleBy
> - DataFrameReader.changes; DataStreamReader.changes; DataStreamReader.xml;
> StreamingQueryManager.close
> - functions: udtf, arrow_udtf, arrow_udf
> - UDFRegistration.registerJavaFunction, registerJavaUDAF
> - ml.connect base classes / param mixins / *Model / ArrayAssembler /
> TorchDistributor
> - Row.count, Row.index
> Module structure: expose pyspark.sql.avro.functions,
> pyspark.sql.protobuf.functions,
> pyspark.sql.functions.partitioning submodules (functions currently exist only
> flat).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
