[
https://issues.apache.org/jira/browse/SPARK-44856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-44856:
-----------------------------------
Labels: pull-request-available (was: )
> Improve Python UDTF arrow serializer performance
> ------------------------------------------------
>
> Key: SPARK-44856
> URL: https://issues.apache.org/jira/browse/SPARK-44856
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 3.5.0, 4.0.0
> Reporter: Allison Wang
> Assignee: Michael Zhang
> Priority: Major
> Labels: pull-request-available
>
> Currently, there is a lot of overhead in the arrow serializer for Python
> UDTFs. The overhead is largely from converting arrow batches into pandas
> series and converting UDTF's results back to a pandas dataframe.
> We should try directly converting Python object into arrow and vice versa to
> avoid the expensive pandas conversion. Similar to this converter:
> [https://github.com/apache/spark/blob/be04ac1ace91f6da34b08a1510e41d3ab6f0377b/python/pyspark/sql/connect/conversion.py#L56]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
