Hyukjin Kwon created SPARK-61157:
------------------------------------
Summary: Column operations capture their call site with a failing
IPython import on every call, which makes them slow without ipykernel
Key: SPARK-61157
URL: https://issues.apache.org/jira/browse/SPARK-61157
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Since SPARK-60915, each DataFrame API call of the Python client captures the
call site in user code through the vendored `_capture_call_site`, and that
function runs `import IPython` and `import ipykernel` on every call. Where
ipykernel is not installed, each import scans sys.path again: 90k Column
operations took 7.3 s instead of 0.32 s with debugging off, and `select` of
2000 names 5 times took 0.52 s instead of 0.006 s. The call site is now
captured in Rust with the same result as `_capture_call_site`, and IPython and
ipykernel are read from sys.modules, with one import attempt per process.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
