[
https://issues.apache.org/jira/browse/SPARK-58072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58072:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid eagerly importing numpy in StatefulProcessorApiClient
> -----------------------------------------------------------
>
> Key: SPARK-58072
> URL: https://issues.apache.org/jira/browse/SPARK-58072
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Structured Streaming, Tests
> Affects Versions: 4.1.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> SPARK-57760 hoisted `import numpy as np` to module top-level in
> `python/pyspark/sql/streaming/stateful_processor_api_client.py` (to define
> `has_numpy` and `_normalize_state_value` once). However, that module is on
> the eager `import pyspark` import chain:
> {code}
> pyspark -> pyspark.sql -> pyspark.sql.context -> pyspark.sql.session
> -> pyspark.sql.dataframe -> pyspark.sql.streaming
> -> pyspark.sql.streaming.stateful_processor
> -> pyspark.sql.streaming.stateful_processor_api_client -> import numpy
> {code}
> As a result, `import pyspark` now eagerly imports numpy, which breaks
> `pyspark.tests.test_import_spark.ImportSparkTest.test_import_spark_libraries`
> (added in SPARK-56242). That test asserts no 3rd party package is imported
> during `import pyspark` to keep import fast:
> {code}
> AssertionError: Unexpected 3rd party package 'numpy' imported during 'import
> pyspark'
> {code}
> This fails several scheduled CI jobs that run the test with numpy installed,
> e.g.:
> - Build / Java25 (Scala 2.13, JDK 25) on master
> - Build / RocksDB as UI Backend (JDK 17) on master
> - Build / Python-only (Minimum dependencies of PySpark) on master
> - Build / Java21 (Scala 2.13, JDK 21) on branch-4.x
> The offending commit is present on both master and branch-4.x, so the fix
> should be backported to branch-4.x as well.
> Fix: resolve `has_numpy` via `importlib.util.find_spec("numpy")` (detects
> numpy without importing it) and import numpy lazily inside
> `_normalize_state_value`. This restores the pre-SPARK-57760 lazy-import
> behavior while keeping the benefit of defining `_normalize_state_value` once.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
