Yicong Huang created SPARK-58923:
------------------------------------
Summary: Bound the output Arrow batch size of Python UDF workers
Key: SPARK-58923
URL: https://issues.apache.org/jira/browse/SPARK-58923
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: Yicong Huang
An Arrow-based Python UDF worker can emit a single oversized output RecordBatch
(for example, applyInPandas produces one batch per group, so a large group
yields one large batch), which strains the JVM and downstream operators. We
need a Python-side (worker-side) mechanism to resize such output batches down
toward a configurable byte limit before they are sent to the JVM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
