[
https://issues.apache.org/jira/browse/SPARK-60306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60306:
---------------------------------
Description: ArrowEvalPythonExec.doExecute and the non-codegen
ColumnarToRowEvaluatorFactory flatten columnar batches into rows without
calling closeIfFreeable (SPARK-50235), so a source that allocates new vectors
for every batch and relies on its consumer to release them keeps them until the
end of the task. As suggested in the review of
https://github.com/apache/spark/pull/59551.
Summary: Release columnar batches with closeIfFreeable in
ArrowEvalPythonExec.doExecute and the non-codegen ColumnarToRow (was: Release
the input batches of Python UDFs with columnar input once they are written to
Python)
> Release columnar batches with closeIfFreeable in
> ArrowEvalPythonExec.doExecute and the non-codegen ColumnarToRow
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60306
> URL: https://issues.apache.org/jira/browse/SPARK-60306
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
> ArrowEvalPythonExec.doExecute and the non-codegen
> ColumnarToRowEvaluatorFactory flatten columnar batches into rows without
> calling closeIfFreeable (SPARK-50235), so a source that allocates new vectors
> for every batch and relies on its consumer to release them keeps them until
> the end of the task. As suggested in the review of
> https://github.com/apache/spark/pull/59551.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
