[
https://issues.apache.org/jira/browse/SPARK-61099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-61099:
----------------------------
Description:
Initial binary result conversion landed in SPARK-61127 (PR #320). This
follow-up implements configuration timing, Arrow-only result conversion, and
iterator failure cleanup in
https://github.com/apache/spark-connect-rust/pull/382.
Eager actions read spark.sql.execution.pyspark.binaryAsBytes after execution
and Arrow-schema conversion, matching PySpark 4.3.0. toLocalIterator()
snapshots the setting on the first next() before execution begins and reuses it
for later rows. The setting also applies when the server supplies only an Arrow
schema.
A configuration or converter-initialization failure terminates the iterator.
Closing an incomplete iterator releases its execution while the native prefetch
task is waiting for another response. Direct binary map keys preserve PySpark's
TypeError when bytearray keys are unhashable.
Regression coverage includes eager aliases, both prefetch modes, absent and
present Spark schemas, empty/null/nested results, configuration failures,
execution ordering and operation-matched execution release. Binary-backed UDT
map-key parity is outside this patch; current-build behavior is not established
by these checks. The branch includes the NumPy scalar-class prerequisite
tracked in SPARK-61102 and preserves connect-rust-4.3.0 release tracking.
was:
Initial binary result conversion landed in SPARK-61127 (PR #320). This
follow-up tracks configuration timing, Arrow-only results, and iterator failure
cleanup.
For eager actions, read spark.sql.execution.pyspark.binaryAsBytes after
execution and Arrow-schema conversion, matching PySpark 4.3.0. For
toLocalIterator(), snapshot the setting on the first next() before execution
begins and reuse it for later rows. Apply the setting even when the server
supplies only an Arrow schema.
A configuration or converter-initialization failure must terminate the iterator
rather than consume a row and allow subsequent iteration to resume. Closing an
incomplete iterator must release its execution even while the prefetch task
waits for another response. Preserve direct binary map-key TypeError behavior
when bytearray keys are unhashable, as in PySpark.
Regression coverage includes eager aliases, both prefetch modes, absent/present
Spark schemas, empty/null/nested results, configuration errors and
operation-matched execution release. Binary-backed UDT map-key conversion is a
separate residual under investigation.
This follow-up preserves the original issue's connect-rust-4.3.0 release
tracking. The implementation is undergoing validation before publication.
> Preserve binaryAsBytes configuration timing and terminate failed iterators
> --------------------------------------------------------------------------
>
> Key: SPARK-61099
> URL: https://issues.apache.org/jira/browse/SPARK-61099
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Initial binary result conversion landed in SPARK-61127 (PR #320). This
> follow-up implements configuration timing, Arrow-only result conversion, and
> iterator failure cleanup in
> https://github.com/apache/spark-connect-rust/pull/382.
> Eager actions read spark.sql.execution.pyspark.binaryAsBytes after execution
> and Arrow-schema conversion, matching PySpark 4.3.0. toLocalIterator()
> snapshots the setting on the first next() before execution begins and reuses
> it for later rows. The setting also applies when the server supplies only an
> Arrow schema.
> A configuration or converter-initialization failure terminates the iterator.
> Closing an incomplete iterator releases its execution while the native
> prefetch task is waiting for another response. Direct binary map keys
> preserve PySpark's TypeError when bytearray keys are unhashable.
> Regression coverage includes eager aliases, both prefetch modes, absent and
> present Spark schemas, empty/null/nested results, configuration failures,
> execution ordering and operation-matched execution release. Binary-backed UDT
> map-key parity is outside this patch; current-build behavior is not
> established by these checks. The branch includes the NumPy scalar-class
> prerequisite tracked in SPARK-61102 and preserves connect-rust-4.3.0 release
> tracking.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
