[
https://issues.apache.org/jira/browse/SPARK-15369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15550012#comment-15550012
]
holdenk commented on SPARK-15369:
---------------------------------
Certainly we can investigate speeding up the serialization between the JVM and
Python as well. I think Wes has some interesting ideas around using Arrow for
something like this (although last I looked the JVM side was maybe a bit far
away from being usable). I'll keep following along with Arrow & related
projects as well :)
The Jython limitations are fairly restrictive its true, but the performance
improvement can be pretty large as well so it might be a reasonable trade-off
for those cases (and also if we eventually no longer have the overhead of
JVM/Python communication be a dominating factor for so many use cases we can
just remove the Jython APIs since the same code should work in regular python).
> Investigate selectively using Jython for parts of PySpark
> ---------------------------------------------------------
>
> Key: SPARK-15369
> URL: https://issues.apache.org/jira/browse/SPARK-15369
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Reporter: holdenk
> Priority: Minor
>
> Transferring data from the JVM to the Python executor can be a substantial
> bottleneck. While Jython is not suitable for all UDFs or map functions, it
> may be suitable for some simple ones. We should investigate the option of
> using Jython to accelerate these small functions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
