[
https://issues.apache.org/jira/browse/SPARK-61157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61157:
------------------------------------
Assignee: Hyukjin Kwon
> Column operations capture their call site with a failing IPython import on
> every call, which makes them slow without ipykernel
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61157
> URL: https://issues.apache.org/jira/browse/SPARK-61157
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Since SPARK-60915, each DataFrame API call of the Python client captures the
> call site in user code through the vendored `_capture_call_site`, and that
> function runs `import IPython` and `import ipykernel` on every call. Where
> ipykernel is not installed, each import scans sys.path again: 90k Column
> operations took 7.3 s instead of 0.32 s with debugging off, and `select` of
> 2000 names 5 times took 0.52 s instead of 0.006 s. The call site is now
> captured in Rust with the same result as `_capture_call_site`, and IPython
> and ipykernel are read from sys.modules, with one import attempt per process.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
