Hyukjin Kwon created SPARK-58073:
------------------------------------
Summary: Avoid eagerly importing numpy in
StatefulProcessorApiClient
Key: SPARK-58073
URL: https://issues.apache.org/jira/browse/SPARK-58073
Project: Spark
Issue Type: Bug
Components: Structured Streaming, PySpark
Affects Versions: 4.2.0, 5.0.0
Reporter: Hyukjin Kwon
SPARK-57760 hoisted {{import numpy as np}} to module level in
{{python/pyspark/sql/streaming/stateful_processor_api_client.py}} (to define
{{has_numpy}} and {{_normalize_state_value}} once). That module is on the eager
{{import pyspark}} chain (pyspark -> sql -> context -> session -> dataframe ->
streaming -> stateful_processor -> stateful_processor_api_client), so it now
imports numpy during {{import pyspark}}.
This breaks {{pyspark.tests.test_import_spark}} (added in SPARK-56242), which
asserts that no 3rd party package is imported during {{import pyspark}}:
{code}
AssertionError: Unexpected 3rd party package 'numpy' imported during 'import
pyspark'
{code}
As a result, the following scheduled jobs fail:
* {{Build / Python-only (Minimum dependencies of PySpark)}} on master
* {{Build / Java21 (Scala 2.13, JDK 21)}} on branch-4.x
The offending commit is present on both master (0af85eee8171) and branch-4.x
(181acbd8db57), so the fix needs to be backported to branch-4.x.
Fix: resolve {{has_numpy}} via {{importlib.util.find_spec("numpy")}} (which
does not import numpy) and import numpy lazily inside
{{_normalize_state_value}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
