Hyukjin Kwon created SPARK-59305:
------------------------------------
Summary: Track Apache Spark 4.3.0: retarget CI to the RC,
re-vendor upstream, and close 4.3.0 client gaps
Key: SPARK-59305
URL: https://issues.apache.org/jira/browse/SPARK-59305
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Fix For: connect-rust-4.3.0
Prepare the spark-connect-rust master branch to track Apache Spark 4.3.0
(currently at RC1).
- Bump tracked version strings (SPARK_VERSION, __version__, PROTO_VERSION) to
4.3.0.
- Retarget the 5 CI workflows to the v4.3.0-rc1 staging distribution
(connect-bundled tarball, no --packages), the v4.3.0-rc1 tag for official
tests, and the staging pyspark sdist.
- Re-vendor the changed upstream Python (pandas, sql/worker, profiler.py,
sql/pandas/serializers.py) to v4.3.0-rc1 and add the new pyspark.messages
package; bump the vendor-drift SPARK_TAG.
- Add new 4.3.0 SQL functions collect_union, to_base32, from_base32; expose
DataFrame.zip in PyO3.
- Docs/examples bumped to 4.3.0.
Follow-ups (need the live 4.3.0 parity environment): variant_strip_nulls bool
arg, TimestampNTZNanos/LTZNanos datatype semantics, golden capture for the new
functions, and the full introspection-diff parity proof.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
