L. C. Hsieh created SPARK-60305:
-----------------------------------
Summary: Interrupting a Python Arrow UDF task can free its input
buffer while it is being written
Key: SPARK-60305
URL: https://issues.apache.org/jira/browse/SPARK-60305
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
Since SPARK-44705, the default (non-pipelined) Python runner serializes the
task input on the task thread into a DirectByteBufferOutputStream inside
BasePythonRunner.ReaderInputStream. The Arrow-based inputs (PythonArrowInput,
ColumnarArrowPythonInput) write that stream through
Channels.newChannel(dataOut), via ArrowStreamWriter and WriteChannel. The
returned channel is an AbstractInterruptibleChannel: each write chunk runs
between begin() and end(), and closing the channel closes the wrapped stream.
If another thread interrupts the task thread while it is inside such a write,
for example Task.kill(interruptThread = true) when a speculative attempt or a
cancelled job is killed, Thread.interrupt() runs the channel's interruptor on
the interrupting thread. That closes the DataOutputStream and therefore the
DirectByteBufferOutputStream, which frees its direct buffer while the task
thread is still copying into it (or copying the old buffer in grow()). This can
crash the executor, silently corrupt memory, or fail with an NPE / internal
error instead of a clean task kill.
Affected: Arrow-based Python UDFs and related operators (pandas/Arrow UDFs,
Arrow-optimized Python UDFs, mapInPandas/mapInArrow, applyInPandas, ...) on the
default non-pipelined path. The pickled BatchEvalPython path writes the stream
directly and is not affected.
Fix: hand the writer a stream whose close() does not close the underlying
buffer, and free the buffer only where the runner closes it itself.
Reported by Dongjoon Hyun in the review of
https://github.com/apache/spark/pull/59332 (SPARK-60113), which fixes the same
issue on the pipelined writer path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
