Hyukjin Kwon created SPARK-61188:
------------------------------------
Summary: Raise RecursionError instead of crashing when progress
handlers run queries without end, in the Python client as PySpark does
Key: SPARK-61188
URL: https://issues.apache.org/jira/browse/SPARK-61188
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
An unguarded progress handler that runs a query (for example
`spark.registerProgressHandler(lambda **kw: spark.range(4).count() if not
kw["done"] else None)`) recurses: the nested query reports progress, which
calls the handler again. Since SPARK-61153 the nested query runs, so the
recursion has no end and the native stack of the process overflows; the
interpreter dies with SIGSEGV (exit 139) and no Python exception. PySpark
raises RecursionError from the nested query (surfaced as
SparkConnectGrpcException "Exception serializing request!") and the process
survives. This change bounds the nesting of progress handlers and hooks that
run queries, and raises RecursionError past the bound, so the process survives
too.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
