L. C. Hsieh created SPARK-60325:
-----------------------------------
Summary: Re-enable Arrow columnar input to Python UDFs, holding
only vectors that are not reused
Key: SPARK-60325
URL: https://issues.apache.org/jira/browse/SPARK-60325
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
SPARK-56350 added an Arrow columnar path to Arrow Python UDFs, which keeps the
pass-through columns of each input batch until its result comes back from
Python, many batches later. SPARK-60294 disabled
spark.sql.execution.arrow.pythonUDF.columnarInput.enabled by default, because a
producer may release or overwrite the vectors it returned before then: e.g. a
reader that refills the same vectors for every batch, the Arrow cache, which
closed the previous root once the next batch was read, or the Python output of
a stacked ArrowEvalPythonExec.
SPARK-60306 added an internal marker, ReusedColumnVector, for vectors that
their producer may release or overwrite after returning them. This ticket makes
the Arrow columnar path hold only vectors without the marker, and take the row
queue path for the others; makes the Arrow cache leave the release of its
batches to their consumer, so it stays on the columnar path; and enables the
configuration by default again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
