[
https://issues.apache.org/jira/browse/SPARK-61100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18140755#comment-18140755
]
DB Tsai commented on SPARK-61100:
---------------------------------
Implementation and validation:
https://github.com/apache/spark-connect-rust/pull/383.
The final patch passed the offline Python suites (2,291 passed; eight explicit
live-server skips), Rust core/API unit tests, formatting and integration-test
compilation with and without the WASM feature. All 12 selected native live
cases and their 36 setup/call/teardown reports passed without selected skips or
failures, with the actual imported extension checked before and after. Eight
smoke actions passed, and introspection confirmed tuple inheritance and no
missing inspected members across 12 API surfaces. The local Python coverage
step passed 2,363 cases with eight skips and reported 84.3% across its 72
configured shim files; that percentage excludes embedded Row source and
compiled Rust.
The scoped official raw-transport comparison recorded 85 passing outcomes with
34 skips, including all 12 mixed Classic/Connect collection cases. Broader full
and cluster raw-transport coverage was accepted through equality of the
selected transport source files with the NumPy prerequisite. Those suites
executed the prerequisite's native binary, rather than this Row binary, and use
official Python converters. Native Row behavior is supported by the separate
native tests and smoke checks. These checks have bounded API, platform and
pandas coverage.
Performance tradeoff: against the final binary parent, fully consumed client
CPU regressed by 31–86%, fully consumed wall time increased by 26–65%, and
retained RSS regressed by about 14–59 MiB for 100,000-row narrow, wide and
nested results. These are three paired samples on one macOS host and one
dependency set. RSS is an endpoint measurement, rather than an allocation,
isolated-peak or leak measurement. The assessment accepts the measured
regression as an engineering tradeoff for tuple compatibility. Official Python
and newest-master performance were not compared. Auxiliary instrumented
profiling timings were excluded from the benchmark results.
Clippy remains an advisory failure (exit 101). Linux and Windows were not run
locally. The prerequisite's separate catalog-plugin drift failure and earlier
failed attempts remain recorded.
Prepared by OpenAI Codex.
> Preserve tuple compatibility for Python Row objects
> ---------------------------------------------------
>
> Key: SPARK-61100
> URL: https://issues.apache.org/jira/browse/SPARK-61100
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Official PySpark Row is a tuple subclass. The Rust-backed Python Row was a
> native extension class, so isinstance(Row(a=1), tuple) was false and
> tuple-aware serializers such as json.dumps could not encode it. Its
> __fields__ accessor also returned a fresh list, losing in-place mutations.
> These behaviors were initially observed on public apache/spark-connect-rust
> master 83d74462ec41012993176cf3af99017576688eb8.
> The proposed implementation in
> https://github.com/apache/spark-connect-rust/pull/383 embeds the PySpark
> 4.3.0 Row implementation and uses one public tuple-based Row identity for
> constructed, collected and nested rows. It preserves tuple operations,
> equality/hash, pickle and mutable field metadata. Collection and iteration
> create eager Rows, with field-name sharing per collected result, per
> top-level iterator Arrow batch and per nested schema path. Local-input and
> UDT conversions use the same public Row identity.
> Regression coverage exercises tuple-aware serialization, field-name mutation
> and sharing, nested inputs, Arrow batches, datatype conversion and
> UDT/spatial results. The inherited customized-pandas named-Row conversion
> behavior remains. The branch includes the NumPy prerequisite in SPARK-61102
> and binary configuration/iterator lifecycle work in SPARK-61099.
> Materialization costs were measured against the final binary prerequisite.
> The measured CPU and retained-memory regressions are accepted as an
> engineering tradeoff for tuple compatibility and are disclosed in the
> validation comment. This issue retains connect-rust-4.3.0 release tracking.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
